Tristan Handy is a lot of things: co-creator of dbt, founder and CEO of dbt Labs, and self-described “startup person.” But besides leading dbt Labs to a

Tristan Handy’s Audacious Vision of the Future of Data Engineering

"We at Yuga are committed to building a Web3 world that's interoperable and fair for all creators, and we're pleased to be collaborating on the OMA and the standards we believe will drive its success,” Yuga Labs CTO Mike Seavers said in a statement.

NFT Creator Royalties Are Broken—Yuga Labs and Magic Eden Join Group Trying to Fix Them

A manufacturer of lab-grown diamonds owned by a Detroit-based private equity firm has moved to liquidate with $48 million in liabilities amid several pending legal disputes, according to a ...

Lab-grown diamond maker with Detroit backing files for bankruptcy with $48M in liabilities

Big data platform startup dbt Labs Inc. said today it’s reinvigorating its flagship dbt Cloud offering with new enhancements and capabilities that it says will enable every employee to ...

Big-data startup dbt Labs rolls out data mesh architecture and enhances semantic layer

SAN DIEGO, Oct. 17, 2023 /PRNewswire/ -- dbt Labs, the pioneer in analytics engineering, has announced the next generation of the dbt Semantic Layer following its acquisition of Transform in ...

dbt Labs Announces the Next Generation of the dbt Semantic Layer, Introduced Alongside New Integration with Tableau

SAN DIEGO, Oct. 17, 2023 /PRNewswire/ -- dbt Labs, the pioneer in analytics engineering, today announced several new product capabilities to its dbt Cloud platform at its annual customer conference, Coalesce 2023. New dbt Cloud capabilities enable ...

dbt Labs Announces Major Enhancements to dbt Cloud to Enable Collaboration at Scale

SAN DIEGO, Oct. 17, 2023 /PRNewswire/ -- dbt Labs, the pioneer in analytics engineering, has announced the next generation of the dbt Semantic Layer following its acquisition of Transform in February 2023. Unveiled during the keynote at dbt Labs' Coalesce ...

SAN DIEGO, Oct. 17, 2023 /PRNewswire/ -- dbt Labs, the pioneer in analytics engineering, today announced several new product capabilities to its dbt Cloud platform at its annual customer ...

The dbt semantic layer is now powered by MetricFlow, the SQL query generation engine obtained from the recent acquisition of Transform Data.

Dbt launches next generation semantic layer to solve trust in data

Get the most recent info and news about dbt Labs on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

dbt Labs is on a mission to empower data practitioners to create and disseminate organizational knowledge. Since pioneering the practice of analytics engineering through the creation of dbt—the open source data transformation tool made for anyone that knows SQL—we've been fortunate to watch more than 5,000 companies use dbt to build faster and more reliable analytics workflows. dbt Labs also supports more than 1,000 customers using dbt Cloud, the centralized development experience for analysts and engineers alike to safely deploy, monitor, and investigate that code—all in one web-based UI.

dbt Labs

Low-code software development, only recently trumpeted as the next big megatrend, is here and it’s already become mainstream. By 2025, 70 percent of new apps developed by organizations will be created using low and no code platform, up from less than 25 percent in 2020 according to . If you’re not already using low-code platforms, you will be very soon. Gartner Low and no-code development makes it possible for people without a technical background to create digital products, which is how the term “citizen developer” came about. However, the idea of the citizen developer is a little misleading. While it is true that you really don’t have to be techy by training to use low-code tools…..some transferable skills definitely help. As data scientist and low-code evangelist, Rosaria Silipo, put it, “ .” Codeless does not mean math-less Furthermore, while organizations and so-called citizen developers are of course benefiting from easier access to digital development, so are technologists. Industry studies show that a significant proportion of professional developers – easily a third and potentially nearly two thirds – are using low-code solutions to speed up their work or create workarounds. Low-Code AI: A Story of Supply and Demand? The only skill that enterprises need even more than IT is data science. The same market forces that are driving up low-code adoption in general are especially acutely felt when it comes to low-code AI solutions. Lack of people is not the only challenge for organizations wanting to get their AI projects up and running. AI development often takes a long time and is resource-intensive when large or complex data sets are involved. In other words, it’s costly. Added to that, competition for data scientists is fierce and remuneration generous. Big data and AI projects involve a serious commitment in terms of money and time. Until now, that is. Low-code is helping to significantly speed up timelines, while bringing down costs not only on a per-project basis but in terms of headcount – there’s mounting evidence that low-code solutions are by reducing the overall number of hires businesses need to make. In the recently updated Emerging Architectures for Modern Data Infrastructure, Andreesen Horowitz noted the ML industry is consolidating around a data-centric approach with low code ML solutions (like Continual and MindsDB) automating the ML modelling process. helping to bridge the skills gap Low-code Tools To Suit Every Job While this might sound like bad news for data scientists, it is worth remembering that data scientists themselves are benefiting from the increased speed and ease of using low-code AI tools. The low-code movement is often described as a “democratization” of all things digital, breaking down the barriers to development and to transformational outcomes. This democratization has been gradually taking place at every level of data science from data input to data transformation to machine learning. There’s a low-code, simpler way of doing many of the jobs that were previously highly complex or took a long time. Using low-code solutions, project timelines speed up and data scientists no longer find themselves locked into tools and processes based on the particular set of programming skills they happen to have. Take SQL as an example, a classic skill and the core language of databases. However, to then work with the data, a data scientist used to need to be proficient in a number of other languages, BI tools and ML solutions. No more: rather than extracting the data from the database, uploading it to a BI tool, later an ML tool, running the models, then loading the findings back into the BI tool, there’s a rapidly growing movement seeking to make the magic happen within the data layer, using the established and universal language of data, SQL. This example, from the field of in-database machine learning, shows how data scientists using standard SQL commands can now query predictions and build ML models from inside the database. Data pipelines are another example of a process that’s being radically simplified and speeded up thanks to companies like with its data transformation product based on SQL commands. dbt Labs Data scientists, who have long debated the relative merits of different programming languages and tools, are increasingly finding humble SQL can get them to their destination quickly and easily. Accelerating Transformation with In-Database Tools It’s an uncomfortable truth for data scientists, but analytics can easily become the bottleneck that slows down digital transformation. There are good reasons behind this – shortage of skills, cost, complexity. Low-code data science tools radically speed up the process of extracting actionable intelligence from data. In the hands of an experienced data pro, the speed, simplicity and cost effectiveness of in-database low-code solutions bring extraordinary possibilities. By making vast amounts of data more accessible, data scientists need only to be constrained by their imagination, rather than the usual reasons things don’t get done (cost, integration issues, time concerns, skills gap). Low-code solutions are helping innovative data scientists shine as they deliver ROI faster and showcase the power of ML for a growing number of scenarios. They also help to increase an organization’s overall digital output. After all, the pandemic caused the world to pivot to digital solutions and there has been no going back. reported an increased demand for new digital products and services, which has continued to grow since. For data and IT teams that were already overworked, overstretch and super stressed, the availability of low-code solutions couldn’t have come at a better time. Three quarters of CIOs The Challenges of a Low-code World Although the adoption of low-code data science tools is rising rapidly, it’s still early days for this type of development. Organizations are understandably worried about the security and compliance implications of non-experts handling data (and other) projects. As development and data science become increasingly decentralized, one of the biggest challenges will be for organizations to foster a culture where innovation can occur freely, but remains strategic, relevant, and secure. Most experts agree that company policies will need to be updated to reflect low-code practices. Low-code AI solutions, when used by data scientists, are a way of working more efficiently, but they could, indirectly, end up adding to their workload. While “citizen development” sounds like it operates independently of the IT team, the reality for data scientists will mean increased calls to help integrate, rescue or otherwise support digital tools that have been created elsewhere. Life as a Low-code Data Scientist Low-code tools represent a major disruption to the way digital has been done so far. Like every disruptive force, it creates opportunities while threatening the status quo. For data scientists, low-code is now becoming a fact of life, a hybrid approach to AI development that should be embraced because of its undeniable speed and simplicity. Furthermore, democratization and decentralization of IT are unstoppable. For data scientists, the question now is not whether to adopt low-code, it is how best to do so. I have seen low code fast-track many data science careers. Understanding their field as they do, they already have an advantage over their non-data colleagues. In the hands of data scientists, low-code solutions help data scientists work faster and smarter while at the same time being more agile and creative.

About The Meteoric Rise of the Low Code Data Scientist

Acerca del ascenso meteórico del científico de datos de código bajo

À propos de The Meteoric Rise of the Low Code Data Scientist

निम्न कोड डेटा वैज्ञानिक के उल्कापिंड उदय के बारे में

ローコード データ サイエンティストの急激な台頭について

Sobre a ascensão meteórica do cientista de dados Low Code

Giới thiệu về sự trỗi dậy số liệu của nhà khoa học dữ liệu mã thấp

关于低代码数据科学家的迅速崛起

There was a time when the data analyst on the team was the person who listened intently to a manager's business hypotheses. data is searched in Excel sheets, queried via email, or queried with SQL. The Data Analyst sets out to find answers by exploring the data for weak signals. One by one, a PowerPoint fills up with graphs and action titles to steer decisions or define a business strategy. Shortly thereafter, the data analyst started an exciting adventure: The Business Intelligence, or BI, analyst was at the forefront of digitization and was a major influence on the impact of data for decision-making. But technological advancements quickly overtook the role of the Data Analyst. A company wants to go faster, bigger and better. Data and consequently Data infrastructure is growing and so are the demands on technical skills: DevOps, DataOps, ML Engineer, Data Engineer, and Full Stack Data Scientist. PowerPoint reports have been replaced by dashboards. The BI analyst role became less defined. While in the past a BI analyst could analyze data in Excel or MySQL and then create a report, now it's just the person who visualizes data, troubleshoots the dashboard, and briefs the IT team. It is the end of the fast, iterative quest to find truth in data. Pipelines are integrated into lengthy sprints and long feedback loops. It's now the job of engineers and not of self-taught Business “McGyver” Analysts. In the worst-case scenario, the Business Analyst becomes the second-class data citizen in the enterprise. Interactions with business stakeholders become theoretical instead of pragmatic and fast. The BI analyst is left out, and the engineering department is left alone and overloaded with tasks. One symptom of this change is moldy data and dashboards. sums it up in his article: Mikkel Dengsøe A dashboard has outdated metrics definitions. A dashboard only partially loads because an underlying data model is broken. KPI targets are imported from an outdated spreadsheet. An exec has bookmarked a dashboard you made in a rush last year and still uses it to make decisions. We need a new understanding of a BI analyst to get back to old strengths. BI Analysts need to be able to build pipelines, test metrics, and visualizations quickly and iteratively despite the Modern Data Stack. The BI Analyst should spend more time on analysis than on fixing dashboard problems. We need an intermediate layer... ... that connects raw data and end metrics. .... that can be easily used by no-coders. ... that integrates with modern data stack. .... that is extensible by engineers with tools they love. You might know Matti and me. We are working a lot with companies on data-driven challenges and trying to find fast solutions and open-source them. . The BI Analysts can build workflows from data blocks (Data Sources = Airbyte under the hood), transformation blocks (Transformation= dbt), and data science blocks and connect them to a dashboard. Ultimately it means that a BI Analyst can fastly iterate metrics and professionalize them in a way (with dbt under the hood) so that an engineer can always extend the tool. with Matti for this very reason that allows BI Analysts to build advanced data pipelines directly through an abstraction layer I have just released an open-source tool This creates a clean codebase under the hood. For example, DBT projects are created and built automatically. We have the idea to build a tool for the data analytics space similar to Webflow for web designers. An experienced engineer can customize the dbt models or create new models and make them available on the Kuwala Canvas for no-coders.  Did I mention we are open-source? 😅 It would be awesome working with you together on a PR. If you are a business - Yes we are still searching for Design Customers! We don’t bite! This flexibility is necessary because data projects are individual, grow, and need to be customizable. Back to topic: Our current project state covers the following points: 💯 The setup of Kuwala is now straightforward. Just clone the repo and pull the dockers! 🗂 You can now easily connect to Snowflake, PostGres, or BigQuery. 🧱 +20 Transformations covering merging, aggregating, and filters for data. Under the hood, we are generating clean dbt Labs models! 🛠 you can now also easily add a new transformation by using dbt-core - here is a guide! You can easily start Kuwala on your local machine over Github, here! We now search for more people using it and building separate parts to grow as a community. We are set - are you? Start hacking! Send us your issues! Start contributing! And if something doesn`t work, and we will help you 🚀 join our Slack community Also Published Here

Eliminating Difference Between Business Intelligence analysts, Data Analysts or Data Scientists 🚀

Eliminando la diferencia entre analistas de Business Intelligence, analistas de datos o científicos de datos 🚀

By Robert Bell, CEO of 2B3D.com "Compassion" is not a word you see in a lot of business plans among top technology and financial firms. It may even be considered something of a liability for most businesses, but I don't see it that way. As the 2B3D team prepares for our upcoming Veterans Day NFT drop, we believe the compassionate aspect of our mission bears repeating: Compassion is our greatest strength, and underpins everything we do. was founded by me as well as my close friend, Andrew Bjarnsen, an Afghan War veteran and combat medic who himself struggles with PTSD. Andrew set out on this mission to help his fellow veterans through the use of virtual reality technology. 2B3D What we are creating at 2B3D (along with our partners at BiocoRx, Mason Labs, and Metabillionaire) is centered around a deep concern for the pain and plight of others. In fact, when we talk about the massive utility and gamification of our NFTs, it is purposely designed in response to the growing need for mental health treatment in a world that needs more empathy all-around- whether in daily 1-on-1 with our neighbors, business partners and fellow citizens, or when we’re thinking about others far removed from our own location and daily experiences. 2B3D is deep in the creation of a first-of-its-kind virtual reality medical environment (VRME) in the metaverse, architected specifically to to deliver free mental health services to military veterans in real-time- and largely to address just about everything that is inadequate about the current status quo for vets seeking these services. We’ll launch this in the first quarter of the new year. Our firm is owned and operated by military veterans, and we intend to head off the growing mental health crisis in America, starting with our at-risk military veterans. We are  deploying new treatments through virtual reality (VRx) that will radically improve the physical and mental health of veterans throughout the nation regardless of their location or distance from big cities. In the last several years, suicide rates have increased, and this burden of self-harm is disproportionately carried by veterans. and we feel something must be done. Veteran suicides rates are vastly outpacing those of the general US population It stands to reason, since resources are strained, the response is too slow. 2B3D’s mission is to build new technologies that can offer solutions. It is our passion. Director Christopher Jones of Public Health Service at the Center for Disease Control and Prevention testified in June that between 2001 to 2019, the rate of suicide In the general population rose nearly 30%, while among veterans the rate increased nearly 36%. What 2B3D is building is a realistic VR medical environment available online 24 hours a day through VR headsets, and in 2D through phones and desktop PCs. Veterans experiencing suicidal ideation (often exacerbated by PTSD symptoms and substance use disorder) will be able to access the 2B3D metaverse for immediate access to suicide prevention counselors, group therapy, meditation spaces, and / treatment. EMDR DBT Our realistic environments will not be just a ‘second-best’ option for suffering veterans, but a cutting-edge solution in an increasingly online society. Veterans who have seen combat or feel isolated are the most vulnerable to thoughts of suicide. 20-30% of veterans who commit suicide which affects divergent thinking, making it nearly impossible to break out of a mental spiral. At those times, seconds literally count. also screen positive for service-related injuries like Post Traumatic Stress Disorder Imagine it's 3 a.m. somewhere in the world where a veteran is suffering a mental health crisis forty-five minutes from the nearest emergency room. They may be considering suicide as an option to ending their pain, and they begin to spiral. Help through 2B3D will be available, always on standby. Rather than leave their home at 3 a.m. to get in a car or find public transportation in search of help, veterans will simply use their VR headset, phone, or computer to access our virtual medical environment equipped with mental health resources. In a safe, virtual space, they also have the option to talk with a live person. Even in the form of avatars, research has shown that this human connection can be enough to prevent a tragedy from occurring. Our funding will be as unique as our service. To that end, we've announced the launch of our Supporter Badge, an NFT collection of 3D PTSD support ribbons that will fund our virtual reality mental health solutions for military veterans. This collection will be available for purchase at this Veterans Day, Nov. 11, 2022. https://mint.2b3d.com By raising funds through the sale of our Supporter Badge NFT collection, we'll be able to provide military veterans with access to proven cognitive therapy modalities in safe and effective VR environments. The public sale price of each 3D ribbon will be $10 dollars. All proceeds go to helping us in our mission to help veterans, and 40% percent of the proceeds will be donated directly to veteran nonprofits and charities also fighting to save veteran lives. Those who are interested in helping veterans do not need experience with NFTs or cryptocurrency to purchase one of our Supporter Badges. They are available for purchase with a credit card on our website. There is no limit to the number of NFTs we can sell to help veterans. For those who love NFTs and helping veterans, the public release of an additional 2,880 unique Collector’s Pass NFTs will also be live on Nov. 11, 2022, with a floor price of .7 ETH each, or approximately $100 until sold out. Every Collector’s Pass is represented as a yet-to-be-revealed 3D object with unique properties like special graphics and rare traits, and future in-game applications. The number of available Collector’s Passes is set at exactly 2,880 - a number representing the 2,880 minutes in the span of two days in which, statistically, 44 veterans will take their own lives. Every minute counts. Join us. Launching Veteran’s Day at . https://mint.2b3d.com ### About Robert Bell Robert Bell is an entrepreneur, investor, and philanthropist who founded 2B3D Inc in 2020 as an all-in-one gaming studio, nonfungible token mint, auction house, and storage system. Bell’s mission was to push the bounds of virtual, augmented and mixed reality technologies as a way to help military veterans suffering from post-traumatic stress disorder. His mission to bring peace and healing to veterans is fueled by play-to-earn games, utility NFTs, and access to exclusive in-game cryptocurrency within a wider decentralized metaverse. Bell is also the founder and CEO of Bell Medical solutions. About 2B3D 2B3D is a military veteran-operated tech company specializing in web3 gaming, virtual reality, NFTs, and mental health solutions in the metaverse. 2B3D is currently developing a metaverse ecosystem using blockchain technology with play-to-earn games, an NFT marketplace and VR showrooms, and VR medical environments for therapy and meeting with healthcare professionals in real-time. The 2880 Genesis Collection NFT will drop on the eleventh day of the eleventh month, in honor of military veterans on Veterans Day. The 2B3D metaverse includes several core projects: VRx | Virtual healthcare with NFT prescriptions and real life professionals. NFTy150 | NFT marketplace, minting option and showroom. Topher's Inferno | Connecting enthusiastic gamers with ambitious developers. RestXP | B2B meeting rooms with a resting crypto reward. So Many Gods | A Play-to Earn, space-themed sci fi looter shooter. Sci-fi To learn more about 2B3D, visit or follow us on Twitter at @2B3Dinc. www.2B3D.com

The Ultimate Utility: 2B3D is Using NFTs to Save Military Veteran Lives

There was an awesome debate on ’s Slack last week discussing mainly two things: DBT The state of open-source alternatives to Fivetran Whether an open-source (OSS) approach is more relevant than a commercial software approach in addressing the data integration problem. If you’re already on DBT’s Slack, here is . Even Fivetran’s CEO was involved in the debate. the thread’s URL In this article, we want to discuss the second point and go over the different points mentioned by each party. The first point will come in another article. It’s more relevant to discuss whether an OSS approach makes sense before drilling down into the different alternatives. We’ll go over the main challenges that companies face and see which approach fits best. We’ll call “commercial companies” the ones with a commercial software product, and “OSS companies” the ones with an open-source approach. 1. Having a large number of high-quality well-maintained pre-built connectors This might be the most challenging part for the open-source approach, but there are actually choices that can make an OSS approach even stronger than a commercial one on that matter. Commercial approach In this case, a company supports a limited number of connectors (the most used ones) and actively maintains them. They know when there is a schema change and when they need to update the connector, and can be pretty responsive on that, if the organization is responsive and scales well. However, the more connectors, the more difficult it is for a commercial company to keep the same level of maintenance across all connectors. In an ideal world, the organization will grow linearly with the number of connectors. But most often, there are inefficient processes, so every organization will reach a limit. The more efficient they are, the higher the limit. Open-source approach When you look at Singer.io, you would think that an OSS approach is at a serious disadvantage here. But Singer.io has actually made several significant poor choices here: There is an absence of standardization / enforcement of protocol, and developers just add whatever they want in their implementation and messages. Integrations live by themselves in their own repos. So each connector can be seen as an individual open-source project. This does not help when building a community. There is no real ownership, nor direction for the project. So can we actually achieve a large number of high-quality well-maintained connectors with an open-source approach? The answer is , but only if the following conditions are met: probably One single repo for the whole project and all connectors. This will help consolidate the developments, and it will unify the community behind one single project and one vision. This means that the project and community can be opinionated about what a connector should be, and how it should be built. Detailed logs for all users, with the ability to easily surface the issue on the GitHub project to the whole community. A separation between extract-load and transformation. Users would be able to use the connector without any normalization, so they use their own transformation processes. By doing so, we can more easily identify where failures happen, and how to solve them. Plus, it would make it easier to support a lot more integrations faster. A last note here: when users use a connector on prod and see it fail, they will be able to fix it without waiting for external support teams (as for commercial companies), and the whole community will be able to benefit from the fix. 2. Addressing unique corner cases and unique needs Every company is opinionated about the way it wants to use a connector; each has specific needs. Commercial approach Commercial companies have a ROI-driven relationship with connectors. Any additional work on any edge case will need to have some ROI, or the company will have a hard time scaling. That also means that the commercial company will most likely have very high quality connectors for the most used ones. But it can’t afford to keep that level of quality across all integrations, and thus won’t be addressing their corner cases. With their finite resources, it will always be a trade-off. Commercial companies can’t address the long tail of integrations and can’t meet all unique needs. Furthermore, in the case a client’s needs are not met, the client won’t be able to customize the existing connector to their needs. That is the limitation of this approach. Open-source approach It is actually not obvious that an OSS approach would easily address all unique corner cases. Contributors build the connectors with their use cases and needs in mind; they won’t be building a high quality connector to address most companies’ needs. However, there are ways to avoid that: Setting standards and enforce protocols, with one single platform / project, as mentioned in the first point. Decoupling extract-load (EL) and normalization. A lot of the customization can be done directly in normalization, but a lot of the EL is common to everyone. In that case, everybody can benefit from the EL part and use their own transformation tools to address their own needs. Enabling contributors to build their connectors using the language of their choice by running connectors as Docker containers Being open source lets any engineer adapt any pre-built connectors to their needs, so every company will be able to address their own needs. The question is more about how much effort would be needed for each of their cases. 3. Debugging autonomy This is an easy one. When dealing with a commercial product, if you face an issue, the only thing you can do is contact their support, and hope that your issue will be prioritized in the list of issues the company’s engineering team must resolve. You won’t be able to know when the issue will be fixed. It can take hours at best, or months. With open source, you don’t need to wait on anybody. You can fix the integration yourself. Having this ability is important if the integration is vital to your business. 4. Getting the internal resources By resources, we mean either budget or the engineering time to work on the integrations. At , we don’t like the word when talking about people. But, in this context. it is the only common one we could think of. Airbyte resources The truth is that in a lot of enterprises, it can be easier to justify hiring a new employee than getting even a modest budget approved for an external vendor. Depending on the open-source project, most or all of the job could be already done. That’s one of the reasons why most companies use open-source technologies rather than external vendors. 5. Easy to use by any teams (analysts, finance teams, etc.) Once consolidated, the data is used by different teams: business intelligence / analysts, data scientists, finance teams, customer success, product or marketing team, etc. That is the beauty of products like Fivetran: anybody can use them and anyone can add more data. One would think that ease of use is exclusive to commercial products, but that is actually not true. For instance, Gitlab and many other OSS projects have UIs that are usable by a non-technical audience. Airbyte has one and it is one of its primary features. 6. Security and privacy compliance We’re talking about data, and most probably this involves customer data. All companies are subject to privacy compliance laws, such as GDPR, CCPA, HIPAA, etc. As a matter of fact, above a certain stage (about 100 employees) in a company, all external products need to go through a security compliance process that can take several months. With open source, teams just use the technology directly without such processes. The adoption is frictionless, and the engineering need can be met overnight. 7. Moving data between internal databases Commercial products mostly sit in the cloud. So if you have to replicate data from an internal database to another, it makes no sense to have the data move through the external vendor. In addition to the vendor’s costs, you need to add egress costs and the issue it poses in terms of privacy. Open-source products are mostly self-hosted, so they don’t have this problem. Is open source hard without a direct revenue channel? One of the comments in the thread was that “open source is hard without a direct revenue channel to support it.” That is actually true for any company. If you have a hard time getting revenues, you can’t scale the team or your effort. According to Bessemer, in revenues than many cloud leaders. open-source companies grow faster Some other interesting quotes / points “The general argument for open-source software that applies to all software: Having access to the source code means users have freedom-the freedom to change the software how they see fit and fix issues.” “If you spent months or years using, say, Fivetran, and all of a sudden Fivetran goes out of business or significantly changes its business, you could run the software yourself, even if just for a period of time. I’ve been there when it’s happened with other software; it’s not pretty, and sometimes you only have days or weeks of heads up.” “If it’s open source, I can run it without Fivetran, so for special circumstances, like compliance or high volume, I don’t have to deal with legal or cost limitations.” “I can see the code running my integration; I can more quickly debug issues. I can also easily create integrations anyone can choose to use. Tools like Fivetran and Stitch only work a percentage of the time; open source allows me to fill in that gap without creating one-off scripts (or Fivetran lambda functions) or going back and forth for weeks or months with support on a bug that I could fix in a few hours.” Conclusion If the open-source project is well structured and well thought through, there will be no points where a commercial approach can be better. The issue is that, until now, there were no good open-source projects that could alleviate the dangers to this approach. If you like how we think an open-source project can address all the possible concerns by building a single platform with a single repo, a defined data protocol, a ready-to-use UI, and decoupling of EL and normalization, you should definitely have a look at what we’re building at . We want to become the open-source standard for data integrations. Airbyte Let us know if you think we missed anything. Our goal is to see things from all perspectives and to keep this article up to date. Also published at: https://airbyte.io/articles/data-engineering-thoughts/open-source-vs-commercial-software-how-to-better-solve-data-integration/

Solving Data Integration: The Pros and Cons of Open Source and Commercial Software  

: Here is not really an article, but more some notes about how we use dbt in our team. We only focus on the data analysis part. Some tools/services are not mentioned here. Note Our team Our technical stack : data storage + development platform (on SQL workspace) BigQuery on : code execution platform, job scheduling, monitoring Airflow Google Cloud Composer : SQL scripts execution dependencies, documentation, and testing dbt client : code versioning, CICD (deployment, code quality, tests) Github : dashboards platform Google Data Studio (or similar): develop code in local, execute tests Visual Studio Code Google Cloud Composer and Google BigQuery are part of Google Cloud Platform (GCP) services. We handle permissions/access by using . We have 2 projects, one for staging and one for production. Identity and Access Management (IAM) Our architecture for analysis Data analysts mainly use SQL scripts to build their analysis. We run every day several hundred SQL scripts. Each script will create a table into BigQuery (with drop/create or incremental strategy). The analysis is split into 3 parts: cleaning raw data cross data and building some common metrics build visualization tables that we can plug into the dashboards (here we use Data Studio) The development process What is compile step? In dbt, you will use the Jinja template inside your SQL code. To evaluate that Jinja template and have only SQL code that you can run on the BigQuery SQL workspace, you need to compile your code Run and test models Inside dbt project, . When you start to have a lot of models, you need to be careful when you run the whole workflow ( without filters). You can quickly consume a lot of resources. each SQL script represents a model dbt run To avoid this, we create custom commands in a makefile. Data analysts are not able to run directly dbt commands, but only makefile commands. It's also easier and less technical for them, they don't have to know dbt commands with all possible parameters. We implemented a “magic method” that runs all models that weren’t tested since the last test execution. That also formats the code and some other stuff. To do it, we store the md5 checksum of each models. By doing this, we are sure that data analysts have tested the code before creating a PR. We also add control on Github CICD (Github actions), to be sure all models are well tested, otherwise the PR is rejected. Documentation format For each model, we create a file that contains the documentation of the model (plus in some cases, tests). .yml This is an example of the documentation file: models:
  - name: user
    description: User of the database (client)
    columns:
      - name: name
        description: First name of the user
        meta:
          sensitive: True
      - name: age
        description: Age of the user
        meta:
          sensitive: True The problem here is that it’s quite hard to be sure that the documentation is up to date with the current model. We develop some utils functions in python that will check the model and compare it with the documentation, it will print differences. It will also automatically generate the documentation file with pre-filled columns. We integrate a check in our Github CICD (Github actions) to be sure that the documentation is up to date, otherwise the PR can’t be merged. Our current feeling about dbt We start to use dbt client for 2 reasons: Handle SQL script dependencies. Before that, we were executing scripts one by one directly on Airflow Have documentation inside the code, and have a nice web UI to explore this documentation. Before that, we were using google spreadsheet (similar to Excel files) to document the code Now we are very happy because we solve those 2 problems without implementing a “homemade solution”. dbt starts to be a standard for this kind of data analysis workflow, and it is always better for the lifecycle of the project to use a standard solution. However, it is important to talk about the less positive points. dbt is a young product, and we saw it when we try to install it on the Google Composer platform. It was very problematic, in fact, the python dbt library has a lot of dependencies, so we had some conflicts with other default Composer libraries. This is not a positive point for the compatibility of the library, and it can very quickly become a constraint. We fix it by creating a virtual env inside the Airflow task, before running dbt. We also realized that most of the development is focused on dbt Cloud, this is normal because it’s a paid solution. But when you use dbt client, it can be frustrating. As an example, we had some difficulties deploying the documentation. Normally, it was supposed to be a static website. But the way how they implement it, you need a web server. To conclude, migration to dbt is not so easy if you don’t start a project from scratch. There are many things to consider. Feel free if you have any technical questions to contact me.

How We Use dbt (Client) In Our Data Team

In this article, I aim to share my experience standardizing the ETL process within frameworks like Spark and Flink, particularly for those not using data processing frameworks like dbt. Context Data processing is a comprehensive field that encompasses the regular processing of substantial information volumes in both batch and stream modes. These procedures, often referred to as ETL processes, are developed using a variety of programming languages and frameworks. Typically, the main stages of data processing are as follows: Data is collected from various sources for subsequent processing. Reading data: The data undergoes various transformations and manipulations to attain the required structure and format. Transformation: The transformed data is then stored in the target storage. Writing: Schematically, it is represented as shown in the diagram: Challenges After the main stages are implemented, several other stages can be added to enhance and refine the process: This stage involves verifying data compliance with requirements and expectations, including data cleaning to ensure accuracy and completeness. Data quality check: The registration process tracks and documents available data sets. It may also involve data and metadata management for effective data organization. Data registration in the data catalog: This allows visualization of relationships between data assets and ETL processes to understand dependencies. It can help identify potential data quality issues and improve overall data quality. Building data lineage: Additional processing may define the level of data confidentiality and criticality, ensuring proper data protection and security. Data classification: Each of the stages mentioned can be implemented using various technologies and tools, allowing flexibility in choosing the right solution for each specific stage of data processing. Solution Reading data The first step in the ETL process involves reading data from the source. Various types of sources may be used at this stage, such as files, databases, message brokers, and more. The data may also come in different formats, such as CSV, JSON, Parquet, Delta, Hudi, etc. Here is some boilerplate code for reading data from these different sources (implemented as Strategy pattern): from abc import ABC, abstractmethod
from pyspark.sql import SparkSession, DataFrame

# Define the strategy interface
class ReadStrategy(ABC):
    @abstractmethod
    def read_data(self) -> DataFrame:
        pass

# File System Read Strategy - represents a file system reading    
class FileSystemReadStrategy(ReadStrategy):
    def __init__(self, spark: SparkSession, path: str):
        self.spark = spark
        self.path = path
    
    @abstractmethod
    def read_data(self):
        pass
    
# Implement the strategy for CSV
class CSVReadStrategy(FileSystemReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("csv").option("header", "true").load(self.path)


# Implement the strategy for Delta
class DeltaReadStrategy(ReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("delta").load(self.path)

# Implement the strategy for Parquet
class ParquetReadStrategy(ReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("parquet").load(self.path)

# Context class
class DataReader:
    def __init__(self, read_strategy: ReadStrategy):
        self.read_strategy = read_strategy

    def read(self) -> DataFrame:
        return self.read_strategy.read_data() Usage example: spark = SparkSession.builder.appName("our-ETL-job").getOrCreate()
path = "path-to-our-data"
src_df = DataReader(DeltaReadStrategy(spark, path)).read() This approach offers several benefits: It enables the alteration of the reading strategy (source format). It facilitates the addition of new reading strategies without modifying existing code. Each strategy can have its unique parameters. For example, CSV may have a delimiter, while Delta may have a version, etc. Instead of providing a path, we can create a dedicated class, like , which can hold all necessary data reading parameters. ReadConfig It allows the incorporation of additional logic into strategies without affecting the main code. For instance: We can call by default for a file source. input_file_name() If we have a data catalog, we can check whether the table exists there and register it if not. Each strategy can be tested individually. Transformation After reading the data, it must be transformed. This can include various operations like filtering, aggregating, and joining. Since the transformation process can be complex and time-consuming, it's crucial to standardize it for ease of understanding. From my experience, using the method of the DataFrame API is an effective way to standardize this process. This method lets us chain multiple transformations, enhancing code readability. Each transformation can be declared as a separate pure function, performing a specific transformation, and can be tested independently. transform import pyspark.sql.functions as F
from pyspark.sql import DataFrame

desired_columns = ["name", "age", "country"]

def with_country_code(df: DataFrame) -> DataFrame:
    return df.withColumn("country_code", F.when(F.col("country") == "USA", "US").otherwise("Other"))

def with_age_category(df: DataFrame) -> DataFrame:
    return df.withColumn("age_category", F.when(F.col("age") < 18, "child").when((F.col("age") >= 18) & (F.col("age") < 65), "adult").otherwise("senior"))

def filter_adult(df: DataFrame) -> DataFrame:
    return df.filter(F.col("age_category") == "adult")

def select_columns(columns: list) -> callable[[DataFrame], DataFrame]:
    def inner(df: DataFrame) -> DataFrame:
        return df.select(columns)
    return inner

transformed_df = (src_df
                  .transform(with_country_code)
                  .transform(with_age_category)
                  .transform(filter_adult)
                  .transform(select_columns(desired_columns))
                  ) Writing The last stage in the ETL process is writing the transformed data into the target storage, which could be a file system, a database, or a message broker. The loading phase may also incorporate extra steps like data validation, data registration, and data lineage. For this stage, we can apply the same methods as reading data, but with writing strategies. from abc import ABC, abstractmethod
from pyspark.sql import DataFrame, SparkSession

class WriteStrategy(ABC):
    @abstractmethod
    def write_data(self, df: DataFrame):
        pass

class FileSystemWriteStrategy(WriteStrategy):
    def __init__(self, spark: SparkSession, path: str, format: str, mode: str = "overwrite"):
        self.spark = spark
        self.path = path
        self.format = format
        self.mode = mode

    def write_data(self, df: DataFrame):
        df.write.format(self.format).mode(self.mode).save(self.path)

class DataWriter:
    def __init__(self, write_strategy: WriteStrategy):
        self.write_strategy = write_strategy

    def write(self, df: DataFrame):
        self.write_strategy.write_data(df) Usage example: path = "path-to-our-data"
writer = DataWriter(FileSystemWriteStrategy(spark, path, "delta"))
writer.write(transformed_df) Conclusion This implementation standardizes the ETL process, making it more comprehensible and maintainable. It facilitates the addition of new reading and writing strategies without modifying existing code. Each strategy can possess unique parameters, and additional logic can be incorporated into strategies without disrupting the main code. The approach also enables separate testing of each strategy. Moreover, we can introduce additional stages to the ETL process, such as data quality checks, data registration in the data catalog, building data lineage, and data classification.

DBT LABS

getdbt.com

201-500 employees

Since 2016

COMPANY RANKING

DBT LABS

EVERGREEN INDEX #109

Representing this company? Click here to claim and customize this page!

dbt Labs WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

DBT LABS

getdbt.com

201-500 employees

Since 2016

COMPANY RANKING

DBT LABS

EVERGREEN INDEX #109

RELATED COMPANIESRANK

STORIES ABOUT DBT LABS

Representing this company? Click here to claim and customize this page!

dbt Labs WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!