Startup Airbyte, creator of the fastest-growing open-source data integration platform, makes moving data easy and affordable across almost any source and destination, helping over 3,000 organizations provide their users with access to the right data for analysis and decision-making

Founded in 2020, Airbyte is the open-source standard for EL(T). We enable data teams to replicate data from applications, APIs, and databases to data warehouses, lakes, and other destinations. We believe only an open-source approach can solve the problem of data integration, as it enables us to cover the long tail of integrations while enabling teams to adapt pre-built connectors to their needs. We've raised $181M from some of the world's top investors (Benchmark, Accel, Altimeter, Coatue, Y Combinator, etc.) and believe in product-led growth, where we build something awesome. We’re continuing to grow with over 25,000 companies syncing data with Airbyte’s 200+ connectors.

Airbyte

How to Chat With Your Data Using OpenAI, Pinecone, Airbyte and Langchain: A Guide

This is a self-contained demo using . Materialize This demo would show you how to use Materialize with Airbyte to create a live dashboard. For this demo, we are going to monitor the orders on our demo website and generate events that could, later on, be used to send notifications when a cart has been abandoned for a long time. This demo is an extension of the tutorial but rather than using Debezium CDC, we are going to use Airbyte to incrementally extract the orders from MySQL over . How to join PostgreSQL and MySQL in a live Materialized view CDC Diagram: Prerequisites Before you get started, you need to make sure that you have Docker and Docker Compose installed. You can follow the steps here on how to install Docker: Installing Docker Note that Airbyte Cloud currently does not support Kafka as a destination, this is why we can only follow this demo with a self-hosted Airbyte instance. Running the Demo Note that for Mac with M1, you might have some issues with the Airbyte due to the following issue: https://github.com/airbytehq/airbyte/issues/2017 So I would recommend using an Ubuntu VM to run the demo. export DOCKER_BUILD_PLATFORM=linux/arm64 
export DOCKER_BUILD_ARCH=arm64 
export ALPINE_IMAGE=arm64v8/alpine:3.14 
export POSTGRES_IMAGE=arm64v8/postgres:13-alpine 
export JDK_VERSION=17 Start all services:

```bash
docker-compose up -d Airbyte Setup the Airbyte service by visiting and then follow the instructions. your_server_ip:8000 Adding a source We are going to use as our source where we will be extracting the orders from. MySQL Via the Airbyte UI, click on the tab and click on the button. Sources Add new source Fill in the following details: Name: orders Source type: MySQL Host: your_server_ip Port: 3306 Database: shop Username: airbyte Password: password Disable SSL Replication Method: CDC Finally, click on the button. Setup source Adding a destination Next, add a destination to Airbyte which will be used to send the events to. For this demo, we are going to use Redpanda but it will work just fine with Kafka. Start by clicking on the tab and click on the button and fill in the following details: Destinations Add new destination Name: redpanda Destination type: Kafka Next, fill up all of the required fields and click on the button. Setup destination Depending on your needs you might want to change some of the settings, but for this demo, we are going to use the defaults. The important things to note down for this demo are: The is Topic orders The is Bootstrap Servers redpanda:9092 Finally, click on the button. Save Set up a connection Now that you have a source and a destination, you need to set up a connection between them. This is needed so that Airbyte can send the events from the source to the destination based on a specific schedule like every day, every hour, every 5 minutes, etc. For this demo, we are going to use a 5-minute schedule. Hopefully, in the future, Airbyte will allow you to customize this and reduce the schedule to 1 minute for example. Click on the tab and click on the button and fill in the following details: Connections Add new connection Set the 'Replication frequency' to 5 minutes Set the 'Destination Namespace' to 'Mirror source structure' Set the source to and the 'Sync mode' to orders Incremental <img width="984" alt="image" src=" "> https://user-images.githubusercontent.com/21223421/158997265-6890282a-a997-495e-b723-265818c8ed24.png Next click on the 'Setup connection' button. And finally, click on the button to start the synchronization. Sync now It might take a few minutes for the connection to be established and the events to be sent. After the synchronization is done, you can see the events in the Redpanda topic that you have specified when you set up the destination. Let's take a look at how to do that! Check the Redpanda topic To check the auto-generated topic, you can run the following commands: Access the Redpanda container: docker-compose exec redpanda bash List the topics: rpk topic list Consume the topic: rpk topic consume orders_topic Note that if you've used a different topic name during the initial setup, you need to change it in the commands above. If you don't see the topic yet, it would be possible that you might have to wait a few extra minutes and also make sure that the service mock is up and running. ordergen Once you've verified that the topic has the CDC events, you can proceed and set up Materialize. Create a Materialize SOURCE Next, we need to create a in Materialize. SOURCE You can do that by heading back to your terminal and running the following commands: Access the container: mzcli docker-compose run mzcli Or if you have installed: psql psql -U materialize -h localhost -p 6875 materialize Create a Kafka by executing the following statement: SOURCE CREATE SOURCE airbyte_source
  FROM KAFKA BROKER 'redpanda:9092' TOPIC 'orders_topic'
  FORMAT BYTES; Note: change to the topic you've specified during the Airbyte setup. orders_topic Use to quickly see the data: TAIL COPY (
    TAIL (
        SELECT
            CAST(data->>'_airbyte_data' AS JSON) AS data
        FROM (
            SELECT CAST(data AS jsonb) AS data
                FROM (
                    SELECT * FROM (
                        SELECT convert_from(data, 'utf8') AS data FROM airbyte_source
                    )
                )
            )
        )
    )
TO STDOUT; You will see a stream of your data as Airbyte sends it to the destination and Materialize processes it with a very minimal, submillisecond delay. For more information on how to use , check out this blog post by : TAIL Joaquin Colacci Subscribe to changes in a view with TAIL in Materialize Create a Materialized View Now that we have a in Materialize, we can create a materialized . A materialized view, lets you retrieve incrementally updated results of your data using standard SQL queries very quickly. SOURCE VIEW To create a materialized view execute the following statement: CREATE MATERIALIZED VIEW airbyte_view AS
  SELECT
    data->>'id' AS id,
    data->>'user_id' AS user_id,
    data->>'order_status' AS order_status,
    data->>'price' AS price,
    data->>'created_at' AS created_at,
    data->>'updated_at' AS updated_at
    FROM (
        SELECT
            CAST(data->>'_airbyte_data' AS JSON) AS data
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT * FROM (
                    SELECT convert_from(data, 'utf8') AS data FROM airbyte_source
                )
            )
        )
    ); Next, run a query to see the data: SELECT * FROM airbyte_view; To visualize the data, you can use a BI tool like or alternatively, as Materialize is Postgres wire-compatible, you can use your favorite programming language and build your own dashboard. For more information on the supported tools and integrations, check out the Metabase Materialized Views documentation Stop the demo To stop the demo, run: docker-compose down -v Useful links Materialize Airbyte Materialize Cloud Materialize demos Redpanda For a similar version of this demo using Debezium, check out the post here: How to join MySQL and Postgres in a live materialized view Community If you have any questions or comments, please join the ! Materialize Slack Community Also Published here

Build a Live Dashboard with Materialize, Airbyte, MySQL and Redpanda/Kafka

At Airbyte, we're very transparent on our journey. We're building an open-source EL(T) platform, I guess it's easier to be fully open and transparent when building open-source technology as it must be part of your DNA. In any case, we hope the lessons from our journey can help other (open-source or not) entrepreneurs in their own journey. Our whole team has had diverse experiences with OKRs (Objectives and Key Results). We’ve seen them implemented in very useful ways, and in some cases in non-productive pressure-inducing ways. In this article, we’d like to share our learnings, as well as Airbyte’s OKRs for Q1 2021. This way, you can see some concrete implementation of OKRs, and, on our side, we can get your feedback and learn from you as well. Why OKRs are powerful for your team and organization Much has already been written on OKRs. If you don’t know about them, here’s a great article: . But we’re sure that many of you are already familiar with OKRs. They are a mechanism for establishing team and company goals, and ways to measure success. OKRs - Secrets to Success In the end, OKRs help us focus our efforts, align our teams, and allow us to track our progress in a meaningful way. They also give each team autonomy for their work. It takes the focus off of creating features and puts it onto creating outcomes for users. That is what makes OKRs so powerful. However, they are just a framework, and you can use them the way you want - which sometimes can be a bit dangerous. OKR pitfalls and how we’re avoiding them Here are a few ways you can lower OKRs’ impact on your team and organization: Too many OKRs or key results. This dilutes the focus and the impact of the team’s focus, and, hence, the outcome. It can be either too many objectives, or too many key results for a said objective. Tasks as key results. It shouldn’t be about the tasks, but the outcome you want from those tasks. You want your team to be able to have the freedom to focus on what they think is most important to achieve the desired outcome. Using them at the individual level can hold some risks. They might be perceived as a means to evaluate,  even if the managers insist they’re not. Not reviewing or updating them when it would be relevant to. Every now and then, the team will learn information that might change their priorities or potential outcomes. In that case, you should revisit your OKRs accordingly. OKRs should always be relevant, or they lose their usefulness. On the 3rd point, we’re not saying that individual OKRs are not useful. Indeed, when you think about the benefits of OKRs, it’s about focus and alignment. Alignment is not achieved at the personal level; however, focus is. But our experience has always been mitigated. We think individual OKRs should be coming from the individuals themselves and should be optional, and not compulsory from the manager. At Airbyte, we’re thinking of company OKRs only. We’re a small team and very horizontal, so no need for departmental OKRs. As we’re early stage, quarterly OKRs are what make the most sense to us, as opposed to annual ones. And here’s how we formulate them. Airbyte’s OKRs for end of Q1 2021 1. O: Growing Community Love What is community love? We like for it. Love is a member's level of engagement and investment in the community. Someone with high love is highly active and plays key roles in the community, like contributing, moderating, and organizing. Orbit’s definition Here are our key results for this objective: KR: Weekly Active Slack users KR: GitHub stars KR: Issue contributors. Issues are a proxy to show that those users are engaging with your product. KR: PR contributors. Pull Requests are the real expression of love and advocacy. Those users dedicated their precious time to your projects. 2. O: Growing Production Usage KR: Active companies, meaning companies that use an active connection that is syncing data at least once a day. KR: Design partners, in order to help us understand everything that is underlying production usage at any company of any stage. KR: Active connections per active companies, as we want this number to grow with time - our goal is that companies will eventually use Airbyte for all their connections. 3. O: Becoming a reliable standard KR: % failure at sync attempts - our goal is to keep it under 5% while growing the use cases we address, throughout Q1 2021. We intend to get that down below 1% by the end of 2021. KR: Response time to any message on Slack or GitHub - our goal is to reach <30 min by end of Q1 2021. KR: Time to high bug resolution - our goal is to reach 1.5 days by the end of Q1 2021. 4. O: Building the Dream Team We strongly believe in , and that it’s better to have one stellar colleague than 5 average ones. talent density KR: 2 A+ engineers KR: 1 founding developer advocate Our next milestones How do these OKRs translate into milestones? Of course, there is a lot more than a simple list of product milestones before you reach your key results. However, we’re taking this opportunity to share with you what we have in mind for the quarter today. Community efforts January: Hard launch on HackerNews once we achieve better connector stability Building tutorials to improve the developer experience (DX) in building their own connectors, or editing pre-built ones Product efforts For our core platform: Integration in data stack with DBT and Airflow Core upgrade strategy For our connectors: Strengthen our connectors so all our connectors are A+ Schemas migration management Seamless OAuth support More high-level abstractions to build connectors more easily (ongoing effort) An MVP for CDC (Capture Data Change) Connector upgrade strategy A public dashboard showing the stability (failure rate) of all our connectors ---- That’s it! We hope the way we use OKRs and how that translates into context at Airbyte was insightful to you. And again, our goal in sharing this with you is also to get your feedback, so don’t hesitate to share your thoughts in the comments. ( : ) Disclaimer The author is the co-Founder at Airbyte

Airbyte's OKRs for Q1 2021

We’ve been asked if Airbyte was being built on top of Singer. Even though we loved the initial mission they had, that won’t be the case. Aibyte's data protocol will be compatible with Singer’s, so that you can easily integrate and use Singer’s taps, but our protocol will differ in many ways from theirs. Let’s first go over the reasons why we don’t build on top of Singer, in contrast with other open-source projects (such as Meltano), and then let’s see how different Airbyte is. Why Airbyte is not built on top of Singer A little history on . It was the first open-source project with the mission to address the data integration problem. It was introduced by the company StitchData (which was acquired by Talend in 2018) as a way to offer extendibility to the connectors they had pre-built. Your company could build their own taps (source connectors). Singer now counts about 150-200 connectors, on par with the closed-source Fivetran. Singer.io Several things: So what is the issue with Singer? 1. Absence of standardization There is an absence of standardization and enforcement of protocol. Developers just add whatever they want in their implementation and messages. Contributors only address their own use cases and needs, and don’t build the connector with the mindset to address most use cases that the community might need. So, you never know the quality of a tap or target until you have actually used it. There is no guarantee whatsoever about what you’ll get. 2. No real ownership There is no real ownership or direction for the project anymore. Indeed, StitchData, over the years, became less and less involved with maintaining the open-source project. And the difficulty with data integration is that applications and APIs change schemas every few months. So a lot of the connectors became outdated, as they were not maintained anymore. In fact, it is not unusual to see connectors with years old PRs that aren’t merged. In the end, you have a In general, the more used a connector is, the more maintained it is. So there is still some value in being compatible with Singer, but building on top of them and being limited by them would not be smart. set of connectors with varying quality. How Airbyte's choices are different from Singer's 1. Airbyte’s connectors are not standalone binaries Singer’s connectors are standalone binaries: you still need to build everything around them to use them. With Airbyte, we want to help take them to the next level with a platform that can orchestrate and make them usable out of the box (through our UI or API). 2. One platform, one project with standards In contrast to Singer, Airbyte has one single repo for the whole project and all the connectors. This will help consolidate the developments, and it will unify the community behind one single project and one vision. This means that the project and community can be opinionated about what a connector should be, and how it should be built. 3. Connectors can be built in the language of your choice Airbyte runs connectors as Docker containers. So connectors can be built in whichever language you want. The overall platform is built using Java, but any team can build their own connector in Python, Go, Javascript, etc. The easier we make contributions to potential contributors, the more active the community will be in building and maintaining connectors. 4. Decoupling of Extract-Load from Transformation A normalization stands for an opinionated view of how one should use the data. By separating extract-load and transformation, Airbyte enables: Engineers, who want to transform the data themselves with their processes, to do that.Engineers / Analysts / Data scientists / Teams, who want to use the normalized data right out of the box, if the normalization is in line with how they want to use the data. It also enables Airbyte to more easily cover the long tail of integrations. Some connectors might not have some normalization, and that can come separately. The community will also be able to contribute their own normalization, so data users can choose which one suits them best. 5. A UI and API to address every teams’ needs Airbyte was built on the premise that a user, whatever their background, should be able to move data in 2 minutes. To do that, we needed to build the UI and also an API. Stitchdata does bring the UI, but it doesn’t come with Singer. Here, Airbyte’s UI and API are both open sourced. Check out on how to move data between Postgres DB in just a few minutes. our tutorial 6. A full commitment to the open-source MIT project with no premium connectors Singer was born after the company StitchData as a way to expand the number of connectors, thanks to the community. StitchData was just selling their product along with it, and was keeping some connectors away from the community as an upgrade lever. Airbyte’s core product is the open-source connectors, and the team is fully committed to expanding and maintaining the connectors. Our doesn’t depend on any premium connectors. Our vision is to become the open-source standard for data integrations, and then to build Enterprise-targeted features (security and privacy compliance, user and role management, support, SLAs, etc.). business model How Airbyte is helping Singer users achieve their goals If you’re using some Singer taps, we’ve got you covered. Our data protocol is not built upon Singer’s, but it is compatible with it. This means we will be able to support Singer’s taps (we will be very selective and focus only on the highest quality and well-maintained ones), and that you can add your own on Airbyte. Note that we will keep very high standards of quality for our connectors, though. So we don’t guarantee we will put forward / support Singer’s low-quality connectors. We hope this article clarifies how is different from (or Meltano, which is built on top of Singer). Our ambitions go beyond what Singer’s protocol can offer. Airbyte Singer Previously published at https://airbyte.io/articles/data-engineering-thoughts/airbyte-vs-singer-why-airbyte-is-not-built-on-top-of-singer/

6 Biggest Differences Between Airbyte And Singer 

When you’re selling or considering purchasing a B2B tool, you need to understand the build vs. buy argument. What are the pros and cons of building the tool internally vs. buying the tool from a third-party vendor? This is especially true in big companies where you have the resources to build the said tools. Early-stage startups will generally opt for the faster route, going with self-served B2B tools -- unless the pricing is prohibitive. But something we don’t often think about is how open-source just messes the whole thing up. The build is completely redefined. You now need to compare the B2B tool with the build without the open-source tool, as well as with the open-source tool, which most often lowers the barrier significantly. In this article, we’ll take the example of the ETL/ELT industry. We know it best, as we’re building Airbyte, the open-source ELT alternative. Let’s see how open-source for ETL / ELT with Airbyte is also flipping the previous Build vs. Buy balance on its head. We’ve produced an infographic to illustrate that point. You will see that without taking Airbyte into consideration, the build vs. buy was pretty useful with Fivetran, in contrast to building connectors yourself. But now, with Airbyte, you can either just use the open-sourced connectors and start replicating data in minutes for free, or even build new connectors (if ever Airbyte doesn’t support them) in a matter of days (vs. months before) with maintenance being crowdsourced throughout the Airbyte community. The Infographic Here is: in white, the original “build” scenario; in blue, the original "buy" scenario with cloud-based Fivetran; in purple, the new "build" scenario with 2 options: “build non-supported connector with Airbyte” in light purple, and “use prebuilt connectors from Airbyte” in dark purple Let’s just say it: the playing field has changed! The Explanation Some context: the average business today uses well over , many of which contain valuable insights about an organization’s operations. Your company is likely on the way to using just as many apps, if not more, and you’ll need a solution to integrate all of the data your apps produce. 100 software apps Time & Effort Building your own pipeline by yourself is a significant time commitment. It can take between 3-6 months to set up a basic pipeline. Furthermore, beyond the time commitment, there is some inherent complexity in building a reliable, high-performance ELT pipeline. You need to: Obtain developer access to the data source Explore the data Design the schema/data models Set up a connector framework Test the connector and validate the data Set up orchestration, configuration validation, state management, normalization, schema migration, monitoring, etc. that happens every few weeks. This part is very cumbersome, as it requires an increasing number of data engineers to manage your connectors. Maintain the connector for every schema change In contrast, an off-the-shelf solution such as Fivetran can be set up in a matter of minutes with prebuilt connectors. Airbyte also takes literally 30 seconds to deploy, and you can start replicating data . within 2 minutes The big difference between both options in terms of time and effort is that all the Fivetran customers we talked to also had to build and maintain connectors on the side, as the connectors they needed were either not supported in the way they needed or not supported at all by Fivetran. That’s where the option to build with Airbyte comes in. For connectors not supported by Airbyte, it is a matter of hours to build connectors. Indeed, Airbyte already took care of having a UI, monitoring, scheduling, orchestration, integration with your data stack, automatic schema changes, etc. There is a very high chance we support your destination. So in the end, it’s only the EL part of the source connector you have to build, and Airbyte is providing some abstractions to make that easier. Regarding maintenance, the goal of Airbyte is to crowdsource throughout the community. When a connector fails because of significant API changes, it will notify the connectors’ users. As soon as the fix is made available by the Airbyte team or a community member, Airbyte will propagate the fix to all the users. The hope is that this approach will provide a better SLA than closed-source solutions such as Fivetran, not to mention the fact that you won’t have to maintain the connector yourself. People & Money From what we’ve seen, a typical company requires the equivalent of at least two or three full-time data engineers to build and maintain a data pipeline. The total cost of three full-time engineers can reach the high six figures (including benefits). So that’s a lot! Fivetran’s fees for a typical mid-sized company with five connectors is about $50,000. But you’ll have to add to that cost all the connectors you need to build and maintain by yourself. In contrast, Airbyte’s connectors are open-sourced, so you can use them for free. You also don’t need to pay for the egress to Fivetran’s infrastructure. It is possible that you might need a little bit of engineering time to operate Airbyte. If you need to build some of the connectors yourself, you will have to pay for the time spent by the data engineering team on building and maintaining them, but that would still be way less than if you had to do everything yourself. Opportunity Costs The actual value brought by your data team is through analysis and modeling. All the data integration, cleaning and transformation is important, as they enable the analysis and modeling. So the more time your team can spend on value-producing tasks, the better for the business. So opportunity costs as depicted in the illustration are very important to consider. Plus, ask any data team -- they will much prefer doing analysis or modeling tasks, rather than pipelining! So you will have better talent retention this way. Now you can see how open-source can flip the previous build vs. buy balance on its head. Before Airbyte, Fivetran was an easy sell. Now, it seems the contrary. Leveraging Airbyte’s open-source technology to build your own data infrastructure seems the obvious choice. There is one last thing to consider when choosing which direction to take: the future. Future Growth of Your Company As your company grows, you will add data sources to the pool. The complexity and effort of building and maintaining a data pipeline for a huge number of data sources can quickly escalate beyond your data engineering team’s ability to handle it. You might consider taking a chance on Fivetran’s ability to cover all or most of your connector needs, so that your team doesn’t need to build and maintain a continually increasing number of connectors (that would defeat the purpose). But, be mindful that Fivetran will always have a ROI consideration to maintaining connectors on the long tail; they won’t maintain connectors that don’t bring enough revenue to offset the maintenance costs. On the other hand, Airbyte will continue to grow the number of prebuilt community-maintained connectors, and can even take a large portion of the maintenance costs off your hands. When making a decision, consider how your company will evolve. And you can be sure that a great data infrastructure that grows with you will be a competitive advantage. And this way of thinking is something that you should have for any tool. Previously published at https://airbyte.io/articles/data-engineering-thoughts/how-open-source-can-disrupt-build-vs-buy-considerations/

Open-source Effect On Build Vs. Buy

Most engineers in their professional life will have to deal with data integrations. In the past few years, a few companies such as Fivetran and StitchData have emerged for batch-based integrations, and Segment for event-based ones. But none of these companies have solved the problem of data integrations, which becomes more and more complex with the growing number of B2B tools that companies use. We don’t think they will ever be able to solve the data integration problem. You might think this is because they are cloud-based and closed source. But fundamentally, we think it boils down to the fact that they don’t aspire for data integrations to become a commodity. However, if you ask engineers, most of them believe data integration pipelines will become a commodity within the next 5 years. And that is our vision at . Airbyte But before we tell you how we intend to commoditize data integration pipelines, let’s review the limitations of current offers. Why is data integration not yet commoditized? 1. Limited number of pre-built maintenance-free connectors When you are closed-source, you need to build and maintain all the integration connectors by yourself. The issue is that this is A LOT of work. It took 6 years for Fivetran to reach 150 connectors that they must maintain every day. And when you consider there are 5,000+ tools in the marktech industry, you understand they will never be able to cover the long tail by themselves. So what happens? Since we started working on this project, we have held interviews with 40 different companies, including a lot of Fivetran and StitchData customers. And a large majority of them had to build some integrations (possibly with Airflow) by themselves to cover the connectors they needed and that were not supported by those tools. In the end, you still have data engineering teams working a lot on building and maintaining integration pipelines, while their expertise could be best leveraged elsewhere. 2. Pricing of cloud-based solutions indexed on volume Another significant issue with the existing solutions is that their pricing is indexed on the volume of data transferred. Because of this, teams need to be careful how they use the connectors. It is super frustrating to have a solution for your integration needs, but be unable to use it the way you need to make your life easier because of a pricing consideration. This is the opposite of what a commodity is supposed to be. 3. Data security and privacy in the Enterprise world There are 2 things that you can be sure will keep growing in the next decades. Companies will leverage more and more data. Companies will need to consider more and more data security and privacy, especially for enterprises. A lot of enterprises have already stopped using 3rd-party cloud-based solutions for security reasons. Those that still use them will all require a lengthy security and privacy compliance process that will last at least 4 months. This cripples internal teams and keeps them from moving forward. A commodity should be easily accessible, and this is currently not the case. What does commoditizing data integrations mean? Here is the world we envision at Airbyte in 5 years: The long tail of connectors should be largely addressed. It should be super easy to build a new connector. There should be built-in scheduling, orchestration, and monitoring for all used connectors. There should be an auto-upgrading mechanism for connectors, so they are maintenance free for data engineering teams. Connectors should be directly in your own cloud, to give companies full control over their data. Connectors could be an extension of your own data infrastructure, as some portability superpowers. There shouldn’t be any cost indexed to the volume of data transferred through the connectors (apart from CPU and egress). Only then can we consider data integration pipelines a solved problem and a commodity. How can we achieve this vision? 1. Open-sourcing all data integration pipelines with a MIT license As mentioned before, data integration pipelines entail a lot of maintenance work. Every tool will see a change in its schemas once in a while. The only way to cover the long tail of connectors is to have a large community of maintainers. But unless you are working for a company whose product is those integration pipelines, you only maintain what you use. That’s why the only way to do that is by open-sourcing those connectors for the larger good through an MIT license. 2. Making building new integrations trivial If ever it wasn’t much simpler to build a new integration using that open-source project, in comparison to building it on the side by yourself, the project would have a harder time finding contributors. The vision would be flawed. That’s why we are focused on making building new integrations trivial. Fortunately, our team has been building data integration pipelines for the past 23 years accumulated, processing more than 100TB of data every day through more than 1,000 integrations. So we know how to build a level of abstraction that will make things easier. And it goes without saying that Airbyte will be providing scheduling and orchestration automatically for your new integrations. Indeed, these 2 things are essential to most teams. 3. Built-in scheduling, orchestration, monitoring, and upgrading In addition to scheduling and orchestration, there are 2 other things we need to provide so connectors are well-maintained throughout the repository: monitoring and a great upgrading experience. Our monitoring needs to give you detailed logs of any error during the data replications, so that you can easily debug by yourself or report an issue to the community, so other contributors can solve it for you. Given that there will be a lot of schema updating for all tools, teams will need to upgrade to the latest version of the repository pretty often in order to ensure they get the updated schemas. 4. Expanding to all types of integrations At the beginning, Airbyte will be focused on batch data replication from 3rd-party tools and databases to warehouses. But nothing prevents us from expanding towards data syncing using warehouses as sources to other destinations, in the near future. For instance, a use case could be if your marketing team wants to send back the data to your ad platforms, so it can better optimize the campaigns. Another use case could be syncing the consolidated data back to your CRM. And later on, we could address the event-based data integrations, a la Segment. Indeed, the technology will be very close to the connectors we will have already built with the community. This would give companies full control over their data in an effortless way. 5. Enabling other data engineering work – transformation, etc. Being open-sourced enables us to go faster and deeper. Compare Gitlab to GitHub, for instance. Gitlab was able to cover a lot more of the value chain. We have that in mind as well with Airbyte. For instance, we are often asked about what we will provide in terms of data transformation. 6. Fulfilling the enterprise requirements with privacy compliance and role management Last but not least, Airbyte will need to address the requirements of enterprises, too. This includes: security and privacy compliance features data quality monitoring features role and user access management, SSO Without these, it will be harder for enterprises to embrace the open-source technology. This is the part that we think we will sell in a source-available enterprise edition. What’s Airbyte’s timeline on this? We’re just getting started on our vision today. We will focus the next year on nailing batch data replication to warehouses. We hope to have at least 50 connectors by the end of 2020, and to be on par with Fivetran by Q2 2021. But we can only achieve that with the help of the community. Right now, you can download our project and self-host it. We built a UI to allow anyone to define their connections and start consolidating data in minutes, just like Fivetran and StitchData. In order to accelerate the release, we built our own version of scheduler so you can get started quickly on a single host. We will very soon integrate with Airflow and Kubernetes so you can dispatch sync tasks across your cluster. Today, our MVP supports BigQuery and Stripe (we wanted to launch fast and get the community  feedback as early as possible). We will add many more sources and destinations in the coming weeks. Give it a spin: . Let us know what you think, and don’t hesitate to star the project if you like our vision to commoditize data integration pipelines! https://github.com/airbytehq/airbyte/ Previously published at https://airbyte.io/articles/engineering-thoughts/how-we-can-commoditize-data-integration-pipelines/

Commoditized Data Integration And How To Achieve It

How we store and manage data has completely changed over the last decade. We moved from an ETL world to an ELT world, with companies like Fivetran pushing the trend. However, we don’t think it is going to stop there; ELT is a transition in our mind towards EL+T (with EL decoupled from T). And to understand this, we need to discern the underlying reasons for this trend, as they might show what’s in store for the future. This is what we will be doing in this article. I’m the co-founder of , the new upcoming open-source standard for data integrations. Airbyte What are the problems with ETL? Historically, the data pipeline process consisted of extracting, transforming, and loading data into a warehouse or a data lake. There are serious disadvantages to this sequence. Inflexibility ETL is inherently rigid. It forces data analysts to know beforehand every way they are going to use the data, every report they are going to produce. Any change they make can be costly. It can potentially affect data consumers downstream of the initial extraction. Lack of visibility Every transformation performed on the data obscures some of the underlying information. Analysts won’t see all the data in the warehouse, only the one that was kept during the transformation phase. This is risky, as conclusions might be drawn based on data that hasn’t been properly sliced. Lack of Autonomy for Analysts Last but not least, building an ETL-based data pipeline is often beyond the technical capabilities of analysts. It typically requires the close involvement of engineering talent, along with additional code to extract and transform each source of data. The alternative to a complex engineering project is to conduct analyses and build reports on an ad hoc, time-intensive, and ultimately unsustainable basis. What changed and why ELT is way better Cloud-based Computation and Storage of Data The ETL approach was once necessary because of the high costs of on-premises computation and storage. With the rapid growth of cloud-based data warehouses such as Snowflake, and the plummeting cost of cloud-based computation and storage, there is little reason to continue doing transformation before loading at the final destination. Indeed, flipping the two enables analysts to do a better job in an autonomous way. ELT Supports Agile Decision-Making for Analysts When analysts can load data before transforming it, they don’t have to determine beforehand exactly what insights they want to generate before deciding on the exact schema they need to get. Instead, the underlying source data is directly replicated to a data warehouse, comprising a Analysts can then perform transformations on the data as needed. Analysts will always be able to go back to the original data and won’t suffer from transformations that might have giving them a free hand. This makes the business intelligence process incomparably more flexible and safe. “single source of truth.” compromised the integrity of the data, ELT Promotes Data Literacy Across the Whole Company When used in combination with cloud-based business intelligence tools such as Looker, Mode, and Tableau, the ELT approach also broadens access to a common set of analytics across organizations. Business intelligence dashboards become accessible even to relatively non-technical users. We’re big fans of ELT at Airbyte, too. But ELT is and has problems of its own. We think EL needs to be completely decoupled from T. not completely solving the data integration problem What’s changing now and why EL+T is the future Merging of Data Lakes and Warehouses There was a great analysis by Andreessen Horowitz about . Here is the architecture diagram of the modern data infrastructure they came up with after a lot of interviews with industry leaders. how data infrastructures are evolving Data infrastructure serves two purposes at a high level: Helps business leaders make better decisions through the use of data — analytic use cases Builds data intelligence into customer-facing applications, including via machine learning — operational use cases Two parallel ecosystems have grown up around these broad use cases. The data warehouse forms the foundation of the analytics ecosystem. Most warehouses store data in a structured format. They are designed to generate insights from core business metrics, usually with SQL (although Python is growing in popularity). The data lake is the backbone of the operational ecosystem. By storing data in raw form, it delivers the flexibility, scale, and performance required for applications and more advanced data processing needs. Data lakes operate on a wide range of languages including Java/Scala, Python, R, and SQL. What’s really interesting is that modern data warehouses and data lakes are starting to resemble one another — both offering commodity storage, native horizontal scaling, semi-structured data types, ACID transactions, interactive SQL queries, and so on. So you might be wondering if data warehouses and data lakes are on a path toward convergence. Will they become interchangeable in a stack? Will data warehouses also be used for the operational use case? EL+T Supports Both Use Cases: Analytics and Operational ML EL, in contrast to ELT, completely decouples the Extract-Load part from any optional transformation that may occur. The operational use cases are all unique in the way incoming data is leveraged. Some might use a unique transformation process; some might not even use any transformation. In regards to the analytics case, analysts will need to get the incoming data normalized for their own needs at some point. But decoupling EL from T would let them choose whichever normalization tool they want. has been gaining a lot of traction lately among data engineering and data science teams. It has become the open-source standard for transformation. Even Fivetran integrates with them to let teams use DBT if they’re used to it. DBT EL Scales Faster and Leverages the Whole Ecosystem Transformation is where all the edge cases lie. For every specific need within any company, there is a schema normalization unique to it, for each and every one of the tools. By decoupling EL from the T, this enables the industry to start covering the long tail of connectors. At Airbyte, we’re building a “ “ so we can get to 1,000 pre-built connectors in a matter of months. connector manufacturing plant Furthermore, as mentioned above, it would help teams leverage the whole ecosystem in an easier way. You start to see an open-source standard for every need. In a sense, the future data architecture might look like this: In the end, extract and load will be decoupled from transformation. Do you agree with us? If so, you might be interested to have a look at what Airbyte does. Previously published at https://airbyte.io/articles/data-engineering-thoughts/why-the-future-of-etl-is-not-elt-but-el/

AIRBYTE

airbyte.com

51-200 employees

Since 2020

COMPANY RANKING

AIRBYTE

EVERGREEN INDEX #495

Representing this company? Click here to claim and customize this page!

Airbyte WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

AIRBYTE

airbyte.com

51-200 employees

Since 2020

COMPANY RANKING

AIRBYTE

EVERGREEN INDEX #495

RELATED COMPANIESRANK

STORIES ABOUT AIRBYTE

Representing this company? Click here to claim and customize this page!

Airbyte WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!