The AI Times is a weekly newsletter covering the biggest AI, machine learning, big data, and automation news from around the globe.

A|I: The AI Times – Google & Nvidia are betting on this Canadian startup to boost ML model performance

In addition, the release will make data more accessible and governed through automatic data audit logging across data sources and will support 14 new data sources including SAP-HANA and Databricks. “Today, everyone can do AI proofs of concept — but ...

Domino Fall 2023 Release Expands Platform to Fast-Track and Future-Proof All AI

Expanded Data Connectivity, New Data Source Audit and Governance Capabilities Help Teams Overcome Barriers to Deploying All AI with Confidence SAN FRANCISCO,  /PRNewswire/ -- Domino Data Lab, provider of the leading Enterprise AI platform trusted by over 20% of the Fortune 100,

Domino Fall 2023 Release Expands Platform to Fast-Track All Enterprise AI, Including GenAI, Responsibly

Machine learning operations startup Domino Data Lab Inc. today announced a major update to its artificial intelligence development platform that it says will make it easier for companies to build generative AI models and other applications.

Domino Data bids to fast-track generative AI model development with project templates

SAN FRANCISCO, CALIFORNIA, UNITED STATES, October 31, 2023 /EINPresswire / -- In a strategic move that promises to redefine the landscape of B2B data solutions, FlashIntel is proud to announce its official partnership with Databricks , marking its esteemed ...

Flashintel Joins Databricks Marketplace: A Partnership Poised To Revolutionize B2B Data Solutions

PORTLAND, PORTLAND, OR, UNITED STATE, October 31, 2023 /EINPresswire / -- Allied Market Research published a new report, titled, " The USD 17 Billion Distributed Cloud Market Reach by 2031 | Top Players such as - Databricks, Axway and VMware." The report ...

USD 17 Billion Distributed Cloud Market Reach By 2031 | Top Players Such As - Databricks, Axway And Vmware

Crisp Acquires Atlas Technology Group from Advantage Solutions to Empower CPG Brands with Better Data

IRVINE, Calif., Oct. 30, 2023 (GLOBE NEWSWIRE) -- Advantage Solutions Inc. (NASDAQ: ADV), a leading provider of outsourced sales and marketing services to consumer goods manufacturers and retailers, has announced the sale of its Atlas Technology Group’s business to Crisp, the open-data retail platform for the consumer goods industry.

Most software leaders will manage generative AI efforts by 2025, Gartner predicts. This trend has implications for their roles that go well beyond application development.

Overseeing generative AI: New software leadership roles emerge

Advanced Analytics Market Use of ML and AI to give customers more personalized experiences, rise of online shopping, and the spread of so

USD 184.4 Billion Advanced Analytics Market Expected To Reach By 2031 | Top Players Such As - Qliktech, Dataiko & Altair

Databricks

databricks

MLOps With Databricks and Spark - Part 1

Building CI Pipeline with Databricks Asset Bundle and GitLab

ডেটাব্রিক্স অ্যাসেট বান্ডেল এবং গিটল্যাব সহ সিআই পাইপলাইন তৈরি করা

Erstellen einer CI-Pipeline mit Databricks Asset Bundle und GitLab

Creación de canalización de CI con Databricks Asset Bundle y GitLab

Création d'un pipeline CI avec Databricks Asset Bundle et GitLab

डेटाब्रिक्स एसेट बंडल और गिटलैब के साथ CI पाइपलाइन का निर्माण

Databricks Asset Bundle と GitLab を使用した CI パイプラインの構築

Databricks Asset Bundle 및 GitLab을 사용하여 CI 파이프라인 구축

Construindo Pipeline de CI com Databricks Asset Bundle e GitLab

Создание конвейера CI с помощью Databricks Asset Bundle и GitLab

Databricks Asset Bundle ve GitLab ile CI Pipeline Oluşturma

Xây dựng quy trình CI với gói tài sản Databricks và GitLab

使用 Databricks Asset Bundle 和 GitLab 构建 CI 管道

If you are using Databricks serving endpoint, and you wish to export metrics to Datadog, you can face with some challenges in Datadog documentation. This article will help you to solve them, at least it will help you to make a PoC. Here are list of links which you can use: https://docs.databricks.com/en/machine-learning/model-serving/metrics-export-serving-endpoint.html https://docs.datadoghq.com/agent/basic_agent_usage/ubuntu/?tab=agentv6v7 Databricks serving endpoint is a service which allows you to deploy your model and serve it. It is a part of Databricks MLflow. You can find more information about it here: https://docs.databricks.com/applications/mlflow/model-serving.html Once you created an endpoint, out of the box you can see some metrics in Databricks UI. Also list of metrics can be received via REST API: curl --request GET "https://${DATABRICKS_HOST}/api/2.0/serving-endpoints/{endpoint_name}/metrics" --header "Authorization: Bearer ${DATABRICKS_TOKEN}" But if you want to export them to Datadog, you need to install and configure Datadog agent. Pre-requisites Created Databricks serving endpoint Bearer PAT token behalf of Databricks user/service principal, which has `CAN_QUERY` permission on the endpoint Datadog API key Enabled OpenMetrics integration in your Datadog account, link: https://app.datadoghq.com/integrations Install Datadog agent Datadog offers several ways to install agent, they are listed here: , however looks like some of them are not working, or at least I was not able to make them work. That is why I decided to use the most simple way — install agent on Docker. https://app.datadoghq.com/account/settings/agent/latest?platform=overview Before agent installation, you need to create a file `datadog.yaml`, which should be located ` `, with the following content: /etc/datadog-agent/datadog.yaml api_key: "__DD_API_KEY__"
site: "datadoghq.com"
hostname: "__YOUR_HOST_NAME__" Then you need to create `metric-config.yaml` file, which should be located ` `, with the following content: /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default instances:
 - openmetrics_endpoint: https://__DATABRICKS_HOST__/api/2.0/serving-endpoints/__ENDPOINT_NAME__/metrics

   metrics:
     - cpu_usage_percentage:
         name: cpu_usage_percentage
         type: gauge
     - mem_usage_percentage:
        name: mem_usage_percentage
        type: gauge
     - provisioned_concurrent_requests_total:
        name: provisioned_concurrent_requests_total
        type: gauge
     - request_4xx_count_total:
        name: request_4xx_count_total
        type: gauge
     - request_5xx_count_total:
        name: request_5xx_count_total
        type: gauge
     - request_count_total:
        name: request_count_total
        type: gauge
     - request_latency_ms:
        name: request_latency_ms
        type: histogram

   tag_by_endpoint: false

   send_distribution_buckets: true

   headers:
     Authorization: Bearer __DATABRICKS_TOKEN__
     Content-Type: application/openmetrics-text The full possible options for ` ` can be found in template file which is located in a container under ` ` conf.yaml.default /opt/datadog-agent/agent/conf.d/openmetrics.d/ I’ve created the next , because I have some difficulties with that one which is provided in Datadog documentation: Dockerfile FROM ubuntu:20.04

# Set environment variables
ENV DD_API_KEY=[SPECIFY_DD_API_KEY]
ENV DD_SITE=datadoghq.com
ENV DD_HOSTNAME=datadog-agent-ubuntu
ENV DATABRICKS_HOST=[SPECIFY_DATABRICKS_HOST]
ENV DATABRICKS_TOKEN=[SPECIFY_DATABRICKS_TOKEN]
ENV ENDPOINT_NAME=[SPECIFY_ENDPOINT_NAME]

# Install curl
RUN apt-get update && apt-get install -y curl

# Copy the datadog.yaml file into the Docker image
COPY datadog.yaml /etc/datadog-agent/datadog.yaml
RUN sed -i -e "s@__DD_API_KEY__@$DD_API_KEY@g" /etc/datadog-agent/datadog.yaml

# Install the Datadog agent
RUN bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

# Copy the metrics-config.yaml file into the Docker image
COPY metrics-config.yaml /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN sed -i -e "s@__DATABRICKS_HOST__@$DATABRICKS_HOST@g; s@__ENDPOINT_NAME__@$ENDPOINT_NAME@g; s@__DATABRICKS_TOKEN__@$DATABRICKS_TOKEN@g; " /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml.default

RUN service datadog-agent restart

# Run a process that doesn't exit
CMD ["tail", "-f", "/dev/null"] Usage Run serving endpoint Define environment variables in `Dockerfile’ Build docker image Run docker container # Step 1: Build the Docker image
# Navigate to the directory containing the Dockerfile
cd /path/to/directory/with/dockerfile

# Build the Docker image
# Replace 'my-image' with the name you want to give to your Docker image
docker build -t datadoc-agent-image .

# Step 2: Run a container from the built image
# Replace 'my-container' with the name you want to give to your Docker container
docker run --name datadog-agent -d datadoc-agent-image

# Step 3: Connect to a container and restart agent
docker exec -it datadog-agent  bash
service datadog-agent restart Some usefully commands for DataDog agent: datadog-agent status
datadog-agent health

service datadog-agent start
service datadog-agent restart Also appears . here

How to Export Metrics from Databricks Serving Endpoint to Datadog

Introduction Databricks is emerging as one of the main players in the MLOps and DevOps world. In the last month, I experienced this as part of my project at Areto and decided to write a series of hands-on blog posts on how to implement a MLOps Pipeline with Databricks. This is an ongoing learning process for me as well. I write each blog post after I experiment with different features and tools. So, I like to see this blog series as collective learning. Any comments and suggestions are welcome. In this blog post, we'll explore how to build a pipeline using the Databricks DBX tool and GitLab. This guide is designed for data engineers and scientists who are looking to streamline their data processing workflows. We'll cover key concepts, practical applications, and step-by-step instructions to help you integrate these tools effectively into your data projects. Continuous Integration (CI) CI/CD (Continuous Integration/Continuous Delivery): Continuous Integration and Continuous Delivery/Deployment (CI/CD) have become fundamental in modern software development practices. At its core, CI/CD is a method that emphasizes short, frequent updates to software through automated pipelines. While traditionally associated with software development, its principles are increasingly being adopted in data engineering and data science. The essence of CI/CD lies in its ability to automate the stages of software development, particularly in building, testing, and deploying code.  Continuous Integration begins with regular code commits to a shared repository branch, ensuring collaborative development without version conflicts. Each commit undergoes an automated build and test process, validating changes to enhance the quality and reliability of the final product. Databricks: Databricks has emerged as a premier cloud-based platform, uniting the realms of data engineering, machine learning, and analytics. Databricks excels in handling large-scale data processing and complex analytical tasks, but to leverage its full potential, teams need to navigate its multifaceted environment effectively. Integrating CI/CD processes within Databricks environments can streamline workflows, ensuring consistent, reproducible, and scalable data operations. This integration is crucial for teams aiming to maintain agility and efficiency in their data projects, enabling them to deliver reliable, high-quality data solutions consistently. Databricks Workflows: Databricks workflows are particularly significant in the context of CI/CD, as they offer a robust platform for automating and orchestrating data pipelines crucial to continuous integration and deployment. These workflows allow for the seamless scheduling and execution of tasks, which is essential for maintaining the frequent, automated update cycles characteristic of CI/CD. In a CI/CD pipeline, Databricks workflows can be used to automatically process and test large datasets, ensuring that data transformations and analyses are consistently accurate and up-to-date. The diversity of tasks that can be executed within Databricks workflows, such as Notebooks for exploratory data analysis, Python scripts for structured data processing, and Python wheels for custom package deployment, aligns well with the varied needs of CI/CD pipelines. By integrating these tasks into CI/CD workflows, data teams can ensure that every aspect of their data processing and analysis is continuously tested and integrated into the larger data strategy. This integration is key for developing resilient, scalable, and efficient data operations, enabling teams to deliver high-quality, reliable data products rapidly. Databricks CLI Extension (DBX): The Databricks CLI Extension, or DBX, is a pivotal tool in integrating Databricks with CI/CD pipelines, enhancing the automation and management of data workflows. The ability to programmatically control and manipulate data processes is crucial to implementing a CI/CD workflow, and DBX fills this role effectively. It provides a command-line interface for interacting with various Databricks components, such as workspaces, workflows, and clusters, facilitating seamless integration into automated pipelines. Hands-on In our hands-on example, we will create a (very) minimal project that uses dbx to deploy and run a Databricks workflow to manipulate, analyze, and test some data. This practical exercise showcases how Databricks can be used for real-world data analysis tasks. Our CI pipeline will play a crucial role in this process, as it will automate the deployment of our code as a Databricks workflow and validate its output. This validation step is vital to ensure the accuracy and reliability of our analysis. By the end of this exercise, you'll have a clear understanding of how to set up and run a Databricks workflow within a CI pipeline and how such a setup can be beneficial in analyzing and deriving insights from large datasets. This example will not only demonstrate the technical application of Databricks and CI principles but also offer a glimpse into the practical benefits of automated data analysis in a business context. Development pattern using Databricks GUI There are various development patterns to implement our CI pipeline. The common development workflows with dbx are to develop, test, and debug your code on your local environments and then use dbx to batch-run the local code on a target cluster. By integrating a remote repository with Databricks, we use a CI/CD platform such as , , or to automate running our remote repo’s code on our clusters. GitHub Actions Azure DevOps GitLab In this tutorial, for the sake of simplicity, we use Databricks GUI to develop and test our code. We follow these steps: Create a remote repository and clone it into our Databricks workspace. We use here. Gitlab Develop the program logic and test it inside the Databricks GUI. This includes Python scripts to build a Python Wheel package, scripts to test the data quality using Pytest, and a notebook to run the Pytest. Push the code to GitLab. The git push will trigger a Gitlab Runner to build, deploy and launch our workflows on Databricks using dbx Git integration As the first step, we . Next, we create a remote repository and . To allow our Gitlab runner to communicate with Databricks API through dbx, we should add two environment variables, and to our CI/CD pipeline configurations. configure Git credentials & connect a remote repo to Databricks clone it to our Databricks repo DATABRICKS_HOST DATABRICKS_TOKEN to generate a Databricks token, in your Databricks, go to User Settings →  Developer → Access tokens → manage → Generate new token. The Databricks host is the URL when you login into your Databricks workspace. It would be something like . The last part of the URL is your workspace id, and you should ignore that. https://dbc-dc87ke16-h5h2.cloud.databricks.com/ Finally we add the token and host to our Gitlab CI/CD setting. For that open your repo in Gitlab, go to Settings→ CI/CD → Variables → Add variables The project skeleton the project is structured into several folders and key files, each serving a specific purpose: .dbx folder: : defining the configuration of your DBX project. It contains environment settings, dependencies, and other project-specific parameters. project.json conf folder: : outlining the deployment configurations and environment settings. It defines workflows and their respective parameters and cluster configurations. deployment.yml : our wheel package. This folder includes: my_package folder A subfolder containing the main ETL task script . The ETL task load our dataset and create two new tables. tasks sample_etl_job.py The file. it includes common utilities that provide access to components such as common.py SparkSession Contains two Jupyter notebooks: notebooks folder: : plot the distribution of different features in our dataset explorative_analysis : used to execute for unit testing run_unit_test pytest This folder is dedicated to testing: tests folder: : includes pytest fixtures conftest.py : contains unit tests to validate the data structure in our tables. test_data.py In addition to these folders, there are two important files in the root directory: : the configuration file for GitLab's Continuous Integration (CI) service, defining the instructions and commands the CI pipeline should execute. .gitlab-ci.yml : for building my Python wheel package. It defines the package's metadata, dependencies, and build instructions. setup.py dbx-tutorial/
├─ .dbx/
│  ├─ project.json
├─ conf/
│  ├─ deployment.yml
├─ my_package/
│  ├─ tasks/
│  │  ├─ __init__.py
│  │  ├─ sample_etl_job.py
│  ├─ __init__.py
│  ├─ common.py
├─ tests/
│  ├─ conftest.py
│  ├─ test_data.py
├─ notebooks/
│  ├─ explorative_analysis
│  ├─ run_unit_test
├─ .gitlab-ci.yml
├─ setup.py Deployment configuration file is crucial for defining your DBX project's configuration. This file could be generated automatically when you run . Check the dbx website for more details about the . project.json dbx --init Profile file reference The “ ” option is used for local development. If you run the command inside the CI tool, you need to specify the and environment variables, and they will the profile variable. Since we don’t want to use the local environment and develop and run our code on Databricks UI, we don’t need to specify this option here. Instead, the dbx will pick them up from the CI/CD setting that we set earlier. profile dbx databricks host databricks token overwrite Your files that dbx will automatically upload to the system will be stored in They will be stored in .  You can read more about this in ml experiment. artifact_location dbx documentation {
    "environments": { 
        "default": {
            "profile": "dbx-tutorial",
            "storage_type": "mlflow",
            "properties": {
                "workspace_directory": "/Shared/dbx/dbx-tutorial", 
                "artifact_location": "dbfs:/Shared/dbx/projects/dbx-tutorial"
            }
        }
    },
    "inplace_jinja_support": false,
    "failsafe_cluster_reuse_with_assets": false,
    "context_based_upload_for_execute": false
} outlines the deployment configurations and environment settings. Here, we use and format to define the workflows. Each workflow defines an object inside the Databricks workspace: or deployment.yml 2.1 API wheel_task job delta live tables Pipeline. custom:
  existing_cluster_id: &existing_cluster_id
    existing_cluster_id: "1064-xxxxxx-xxxxxxxxx"

environments:
  default: #this should be same envoriment name as you defined in the project.json
    workflows:
      - name: "etl_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            python_wheel_task:
              package_name: "my_package"
              entry_point: "etl_job" # take a look at the setup.py entry_points section for details on how to define an entrypoint
          - task_key: "eda"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/explorative_analysis"
              source: WORKSPACE
            depends_on:
              - task_key: "main"    
      - name: "test_job"
        tasks:
          - task_key: "main"
            <<: *existing_cluster_id
            notebook_task:
              notebook_path: "/Repos/<your DB username>/dbx-tutorial/notebooks/run_unit_test"
              source: WORKSPACE
            libraries:
            - pypi:
                package: pytest Here, we define two workflows: : consists of two tasks/jobs. "etl_job" The first task ( ) is of type . It used to run ETL task using the entry point that we define in the file. main python_wheel_task setup.py The second task ( ) is of type . It is used to run the notebook after the main tasks run successfully. Notice the use of property. etl notebook_task explorative_analysis depends_on : consists of only one job of type . It is used to run the notebook that is "test_job" notebook_task responsible for running the pytest. : in the above example, we run the workflow on an existing cluster. You can also create and run a new cluster for every deployment and launch of your work. For that, you should change the code as follows: Note custom:
  basic-cluster-props: &basic-cluster-props
    spark_version: "11.3.x-cpu-ml-scala2.12"

  basic-static-cluster: &basic-static-cluster
    new_cluster:
      <<: *basic-cluster-props
      num_workers: 1
      node_type_id: "Standard_E8_v3"

environments:
  default: 
    workflows:
      - name: "et_job"
        tasks:
          - task_key: "main"
            <<: *basic-static-cluster
						.... CI pipeline configuration Your GitLab CI pipeline is structured to automate the testing and deployment processes of your Databricks project. It consists of two main stages: and . In the stage, the runs the unit tests and deploys a separate workflow for testing. The stage, activated upon successful completion of the test stage, handles the deployment of your main ETL workflow. In general, the pipeline follows these steps: test deployment test unit-test-job deploy Build the project Push the build artifacts to the Databricks workspace Install the wheel package on your cluster Create the jobs on Databricks Workflows Run the jobs image: python:3.9

stages:          # List of stages for jobs, and their order of execution
  - test
  - deploy

unit-test-job:   # This job runs in the test stage.
  stage: test    # It only starts when the job in the build stage completes successfully.
  script:
    - echo "Running unit tests... This will take about 60 seconds."
    - echo "Code coverage is 90%"
    - pip install -e ".[local]"
    - dbx deploy --deployment-file conf/deployment.yml test_job --assets-only
    - dbx launch test_job --from-assets --trace

deploy-job:      # This job runs in the deploy stage.
  stage: deploy  # It only runs when *both* jobs in the test stage complete successfully.
  script:
    - echo "Deploying application..."
    - echo "Install dependencies"
    - pip install -e ".[local]"
    - echo "Deploying Job"
    - dbx deploy --deployment-file conf/deployment.yml etl_job
    - dbx launch etl_job --trace
    - echo "Application successfully deployed."
    - echo "remove all workflows." For workflow deployment and lunch, we use two commands: : to deploy the workflow definitions to the given environment. dbx deploy : to launch the given workload by its name in a given environment. dbx lunch In the we use the flag to avoid creating a job definition on our Databricks workflows during deployment. For the lunch command, we should use the flag. If this flag is provided, the launch command will search for the latest deployment that was assets-only. Lunching this, we can not see anything in the Workflows UI, but it will be a non-titled job  running by submit API. You can see this on the Job Runs page **** of your Databricks workspace. Check out the dbx documentation to read about and unit-test-job --assets-only --from-assets assest-based workflows other commands If you check the “Experiments” page of your Databricks workspace, you will find the build artifacts of each run of your CI pipeline. Databricks Asset Bundles currently, Databricks recommends using Asset Bundles for CI/CD and offers a migration guide from dbx to bundles.  If you understand the concepts in this post, it would be easy to switch to bundles. In the next post, I am going to explain how we can convert this project to an asset bundle project. Resources for this tutorial. Make sure you update the file with the correct and Repository conf/deployment.yml cluster_id DB username dbx documentation Databricks Best Practices: Development loop and CI/CD on Databricks with dbx Databricks Jobs API CI/CD techniques with Git and Databricks Repos What is CI/CD on Azure Databricks?

Building a CI Pipeline with Databricks dbx Tool and GitLab

In this post, we will look at creating an Azure data factory with a pipeline that loads Office 365 event data incrementally based on change data capture (CDC) information in the source of Change Data Feed(CDF) of a Delta lake table to an AWS S3 bucket. What we’ll cover: Create an ADF Pipeline that loads Calendar events from Offfice365 to a Blob container. Run a Databricks Notebook with the activity in the ADF pipeline, transform extracted Calendar event and merge to a Delta lake table. Load Change Data Feed on the Delta lake table to an AWS S3 bucket. The following diagram shows the architecture of the above steps: Prerequisites To build the pipeline, there are a few prerequisites that need to be met: Setup an Azure account. Register an AD application with Microsoft Graph permission. . Configure Microsoft Graph data connect consent request approver group t. Create an Azure Storage Accoun Create an AWS S3 bucket. Before going any further, we will need to create an Azure Storage account and grant the Azure AD application we registered the proper access to the container of the storage account. Check out the step-by-step Azure for Authorizing access to blobs using Azure Active Directory. guide 1. Create an Azure Data Factory Pipeline In this step, we will create an ADF Pipeline that loads Office365 Calendar Views data to an Azure Blob container. I will assume you already have an Azure Subscription and Resource Group setup. To create an Azure data factory, run the following command: az datafactory create --resource- --factory-name ADFDemo group RESOURCE_GROUP 1.1 Create linked services and datasets To connect to Office 365, we need to use the following command to create a linked service for Office 365 connection: az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name Offic365LinkedService - - properties @offic365LinkedService . json Replace APP_CLIENT_ID , TENANT_ID and APP_CLIENT_SECRET with corresponding values from your Active Directory tenant and App. Next, create a Calendar dataset named calendarDataset to reference the Calendar view data from Office 365. az datafactory dataset create - - resource - group RESOURCE_GROUP - - dataset - name calendarDataset - - factory - name ADFDemo - - properties @calendarDataset . json Now, let’s create another Linked Services for Blob Storage Account to store calendar data extracted from Office 365. az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name AzureStorageLinkedService - - properties @azureStorageLinkedService . json ECRET Replace YOUR_CONTAINER , APP_CLIENT_S , TENANT_ID and APP_CLIENT_SECRET with corresponding values from your Active Directory tenant, App, and storage account. Next, create a binary dataset named . outputDataset az datafactory dataset create - - resource - group RESOURCE_GROUP - - dataset - name outputDataset - - factory - name ADFDemo - - properties @outputDataset . json Note that @dataset.outputPath in the folderPath section is a global variable that we will set in the next step. 1.2 Create an ADF pipeline In this section, we will use the Azure CLI command to create a simple ADF pipeline with three activities: to set a global variable . SetVariableActivity outputPath named to load data from Office 365 to Azure blob storage. CopyActivity ExtractCalendarView to throw an error when the pipeline fails. FailActivity Enter the following commands to create ADF Pipeline with activities: az datafactory pipeline create --resource-group RESOURCE_GROUP --factory-name ADFDemo --name DemoPipeline  --pipeline @demoPipeline.json Once the pipeline is created, Navigate to the Azure portal and launch the Azure Data Factory studio. We can view the pipeline activities we have created from the pipeline designer. 2. Create Notebook in Azure Databricks In this section, we will create a notebook in Azure Databricks. You can refer to this to create a notebook in the Azure Databricks workspace. tutorial Before starting the notebook, we have to set up a connection from Azure Databricks to Azure Storage Account using a SAS key. Check out this for creating a Databricks secret scope. tutorial We are going to perform the following steps in the notebook: Mount a Blob storage container. Transform calendar view data. Create an SCD Type 1 delta table with Change Data Feed enabled. Merge input data frame to the delta table. Configure AWS S3 connection, and load change feed data to s3 bucket. The notebook would look similar to this: Now we have our notebook created, Let’s move back to step 1.1 to create linked services for Azure Databricks connection. az datafactory linked - service create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - linked - service - name databricksLinkedService - - properties @databricksLinkedService . json Next, delete the pipeline that we created in step 1.2, recreate it to execute the Databricks notebook against the Databricks jobs cluster. Delete the existing pipeline we have created: az datafactory pipeline delete - - factory - name - - name ADFDemo - - resource - group RESOURCE_GROUP then recreate the pipeline using the following command: az datafactory pipeline create - - resource - group RESOURCE_GROUP - - factory - name ADFDemo - - name DemoPipeline - - pipeline @demoPipeline . json Now we should have our pipeline that looks like the following diagram. Let’s publish entities (linked services and pipeline) to the Azure Data Factory service using ADF studio. and trigger the pipeline. We can log on to the Azure Databricks workspace, go to Clusters and see the Job status as . pending execution, running, or terminated After the pipeline run is completed, log on to the AWS S3 console, we should see the CSV file in the following format is loaded to the destination s3 bucket. Conclusion In this article, I introduced the approach to load data incrementally from Office365 to an AWS S3 bucket using ADF and Azure Databricks. I hope you found this article useful. You can find the ARM template and notebook in my . GitHub repository

Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using ADF and Databricks

Introduction The Data space is almost as volatile as the AI space this year, with many players consolidating. In the summer of 2022, I wrote about The Data Table Format Wars, and the ecosystem has only gotten more interesting since then. I won’t get deep into the weeds on Data Lakes vs. Data Warehouse, etc., but suffice it to say, we are seeing everything coalesce around decoupling data storage from computing. Table formats are a way to standardize lightly opinionated storage. As I opined a couple of years ago, Apache Iceberg seems to be where they are settling. These formats tend to use Apache Parquet on object storage and provide a method for tracking new data and data state changes. In 2023, Databricks announced the open-sourcing of their Unity Catalog, and Onehouse announced OneTable (now X-Table). These projects are meant to give you a single interface to the three popular table formats: Delta, Hudi, and Iceberg. Then, June 2024 happened… What Happened? Snowflake was having its annual Snowflake Summit conference, and on June 3, 2024, they announced Apache Polaris (incubating), a vendor-neutral, open catalog for Apache Iceberg. The next day, and a week before the Databricks conference, Databricks announced their acquisition of Tabular, a company founded by the creators of Iceberg. This still seems weird to everyone from the outside because Iceberg is a spec; it isn’t a product, and Tabular isn’t even the main contributor to Iceberg. In my opinion, Databricks will somehow roll their Deltalake into Iceberg. Behind the scenes, Snowflake had been trying to acquire Tabular. Something happened at the 11th hour, and Databricks came in and laid down a ton more money, and the Snowflake bid was discarded. Snowflake was already two years into their commitment to supporting Iceberg as external storage, which made a lot of sense for them regardless. Snowflake has a reputation concerning the release speed after an announcement, so there was some skepticism, and some even wondered if any code had been written. It turns out Snowflake had been working on this for a while, and it was made available in very short order, with vendors like Dremio, Starburst, CelerData, and Upsolver showing solutions using it in less than two months: a delightful surprise, and a positive sign of things to come. All of that is a long way to set up Apache Polaris, and hopefully, it will give you a sense of the importance that a good catalog plays in all of this. How will any of your compute tools work if you don’t have a unified system to describe all those parquet files in your data lake? Polaris Overview Apache Polaris is an open-source catalog implementation for Apache Iceberg tables. Its main features include: REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines.
Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog.
Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines.
Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables.
Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage.
Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution.
Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality.
Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs.
Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place.
Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. And a lovely little graphic from the docs to put a picture on everything. Borrowing from the Polaris docs for a picture once again, let’s dive into what is happening here in this entity hierarchy graphic: We can have multiple catalogs in a single Polaris install; in this diagram, it is only Catalog1 that has any namespaces or tables defined. A single catalog can be either internal or external. The internal tables allow for read/write functionality, meaning you have full control. The external tables are currently read-only and are intended to mirror data that is in some external catalog, such as a Glue catalog, which Polaris will see as the source of truth once connected. The catalog organizes your iceberg tables; you’ll want to configure it with your storage configuration for the applicable cloud system that you are on. This forms the first architectural layer in the Iceberg table spec. A namespace can have an arbitrary level of nesting and is used to logically group Iceberg tables within a catalog. An Iceberg table will always belong to a namespace. With this configuration in place, you can then use the Polaris catalog to describe your Iceberg tables in such a way as to make them easily accessible and manageable using a large variety of tools. That’s a quick overview of what is happening and what it looks like. The implementation inside of Snowflake provides a really handy UI to manage the implementation; other vendors have their own, and you can just do it through Spark if you like. Summary I continue to believe that Apache Iceberg is going to “win” the table format wars. Apache Polaris focusing on Iceberg is not a bad thing. I want to dig into how RBAC has been implemented in Polaris another time and see if it is as robust as what the Tabular product implemented, which I was a big fan of. I believe Polaris is a big win for the Iceberg community and will help accelerate adoption even further. I’ve seen talk about the Dremio Nessie project getting rolled in, and I’m curious to see what that will look like. I haven’t looked at Nessie in a couple of years, and I thought their git-like semantics mainly had been covered by the Iceberg tagging and branching, which I wrote about last year—yet another story for another day. I’m a big fan of what is happening here. If you are in this space, this is the right train to jump on. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB?
What the Heck Is Malloy?
What the Heck is PRQL?
What the Heck is GlareDB?
What the Heck is SeaTunnel?
What the Heck is LanceDB?
What the heck is SDF?
What the Heck is Paimon?
What the Heck is Proton?
What the Heck is PuppyGraph?
What the Heck is GPTScript?
What the Heck is WarpStream? Introduction Introduction The Data space is almost as volatile as the AI space this year, with many players consolidating. In the summer of 2022, I wrote about The Data Table Format Wars , and the ecosystem has only gotten more interesting since then. I won’t get deep into the weeds on Data Lakes vs. Data Warehouse, etc., but suffice it to say, we are seeing everything coalesce around decoupling data storage from computing. The Data Table Format Wars Table formats are a way to standardize lightly opinionated storage. As I opined a couple of years ago, Apache Iceberg seems to be where they are settling. These formats tend to use Apache Parquet on object storage and provide a method for tracking new data and data state changes. In 2023, Databricks announced the open-sourcing of their Unity Catalog, and Onehouse announced OneTable (now X-Table). These projects are meant to give you a single interface to the three popular table formats: Delta, Hudi, and Iceberg. Then, June 2024 happened… announced announced What Happened? Snowflake was having its annual Snowflake Summit conference, and on June 3, 2024, they announced Apache Polaris (incubating), a vendor-neutral, open catalog for Apache Iceberg. The next day, and a week before the Databricks conference, Databricks announced their acquisition of Tabular, a company founded by the creators of Iceberg. This still seems weird to everyone from the outside because Iceberg is a spec; it isn’t a product, and Tabular isn’t even the main contributor to Iceberg. In my opinion, Databricks will somehow roll their Deltalake into Iceberg. announced announced Behind the scenes, Snowflake had been trying to acquire Tabular. Something happened at the 11th hour, and Databricks came in and laid down a ton more money, and the Snowflake bid was discarded. Snowflake was already two years into their commitment to supporting Iceberg as external storage, which made a lot of sense for them regardless. Snowflake has a reputation concerning the release speed after an announcement, so there was some skepticism, and some even wondered if any code had been written. It turns out Snowflake had been working on this for a while, and it was made available in very short order, with vendors like Dremio, Starburst, CelerData, and Upsolver showing solutions using it in less than two months: a delightful surprise, and a positive sign of things to come. All of that is a long way to set up Apache Polaris , and hopefully, it will give you a sense of the importance that a good catalog plays in all of this. How will any of your compute tools work if you don’t have a unified system to describe all those parquet files in your data lake? Apache Polaris Polaris Overview Apache Polaris is an open-source catalog implementation for Apache Iceberg tables. Its main features include: REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines. Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog. Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines. Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables. Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage. Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution. Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality. Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs. Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place. Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. REST API implementation: Polaris implements Iceberg's REST API, enabling interoperability across multiple query engines. Multi-engine compatibility: It allows various query engines, such as Apache Spark, Apache Flink, Trino, and others, to read and write Iceberg tables through a single catalog. Centralized management: Polaris provides centralized and secure access to Iceberg tables across REST-compatible query engines. Namespace and metadata management: It supports creating namespaces for logical grouping and manages metadata for Iceberg tables. Storage configurations: Polaris offers robust storage configurations for cloud providers like S3, Azure, and Google Cloud Storage. Security features: It implements role-based access control (RBAC) and uses credential vending to secure query execution. Open-source nature: Polaris is fully open-source, allowing for community contributions and vendor neutrality. Interoperability: It enables multiple engines to be used on a single copy of data, reducing storage duplication and ETL costs. Consistent access control: Polaris allows you to manage role-based access control and storage layer security consistently across engines from one place. Flexibility in hosting: It can be hosted on various infrastructures, providing flexibility and reducing vendor lock-in. And a lovely little graphic from the docs to put a picture on everything. Borrowing from the Polaris docs for a picture once again, let’s dive into what is happening here in this entity hierarchy graphic: We can have multiple catalogs in a single Polaris install; in this diagram, it is only Catalog1 that has any namespaces or tables defined. A single catalog can be either internal or external. The internal tables allow for read/write functionality, meaning you have full control. The external tables are currently read-only and are intended to mirror data that is in some external catalog, such as a Glue catalog, which Polaris will see as the source of truth once connected. The catalog organizes your iceberg tables; you’ll want to configure it with your storage configuration for the applicable cloud system that you are on. This forms the first architectural layer in the Iceberg table spec. A namespace can have an arbitrary level of nesting and is used to logically group Iceberg tables within a catalog. An Iceberg table will always belong to a namespace. With this configuration in place, you can then use the Polaris catalog to describe your Iceberg tables in such a way as to make them easily accessible and manageable using a large variety of tools. That’s a quick overview of what is happening and what it looks like. The implementation inside of Snowflake provides a really handy UI to manage the implementation; other vendors have their own, and you can just do it through Spark if you like. Summary I continue to believe that Apache Iceberg is going to “win” the table format wars. Apache Polaris focusing on Iceberg is not a bad thing. I want to dig into how RBAC has been implemented in Polaris another time and see if it is as robust as what the Tabular product implemented, which I was a big fan of. I believe Polaris is a big win for the Iceberg community and will help accelerate adoption even further. I’ve seen talk about the Dremio Nessie project getting rolled in, and I’m curious to see what that will look like. I haven’t looked at Nessie in a couple of years, and I thought their git-like semantics mainly had been covered by the Iceberg tagging and branching, which I wrote about last year—yet another story for another day. wrote about I’m a big fan of what is happening here. If you are in this space, this is the right train to jump on. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the heck is SDF? What the Heck is Paimon? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is GPTScript? What the Heck is WarpStream? What The Heck Is DuckDB? What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is SeaTunnel? What the Heck is LanceDB? What the Heck is LanceDB? What the heck is SDF? What the heck is SDF? What the Heck is Paimon? What the Heck is Paimon? What the Heck is Proton? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is PuppyGraph? What the Heck is GPTScript? What the Heck is GPTScript? What the Heck is WarpStream? What the Heck is WarpStream?

What The Heck is Apache Polaris?

This summer both Databricks and Apache Iceberg rolled out enhancements to their open table formats. Databricks announced can read and write data to all of the most popular open table formats: Delta Table, Iceberg and Apache Hudi. Delta Universal Format (UniForm) makes it so that open table formats become interoperable, avoiding the need for creating and storing extra copies of data in this or that format. Data teams using existing query engines like , , and others to query over Iceberg or Hudi files can read Delta tables directly without conversion. Delta Lake 3.0 DuckDB Dremio Around the same time, Iceberg announced a slew of new support for query engines and platforms including , , and StarRocks. With these announcements from Databricks and Iceberg, interoperability joined hand-in-hand with data portability. Open table formats by design promote the concept that you should be able to access, control, share, and operate on your data with whatever tool you want, wherever you want to, whether that is in the public clouds, in your private clouds, on-edge, or on bare-metal. Snowflake AWS Athena Apache Doris Understanding Open Table Formats Let’s put these announcements in context. Open table formats allow data lakes to attain performance and compliance standards that in the past could only be achieved by traditional data warehouses or databases, all the while preserving the flexibility of a data lake environment. There are three major open table formats: was originally designed by Netflix specifically for handling substantial data volumes within data lakes. This open table format boasts distinctive features such as time travel, dynamic schema evolution, and partition evolution. These capabilities make it revolutionary, enabling concurrent and secure operations by query engines on the same dataset. Iceberg is an open-source storage framework in the Lakehouse architecture that empowers data lakes on object storage like MinIO. It ensures ACID transactions, scalable metadata handling, and unified processing for Apache Spark, offering reliability and scalability. Delta Lake can handle the performance and correctness challenges of complex Spark workloads, especially under heavy concurrency, with non-atomic updates and metadata operations causing significant bottlenecks. Delta Lake is rooted in the Hadoop ecosystem, and Hudi's primary purpose is to decrease latency during the ingestion of streaming data, offering features such as tables, transactions, upserts/deletes, advanced indexes, and compatibility with various storage implementations, including cloud-native object storage like MinIO. Hudi Much has been written about choosing between different formats, with some asserting up to among the three primary Open Table formats. This blending of distinctions makes sense given the environment of interoperability in which these open table formats were created and continue to thrive. The creators of these formats prioritized capability over traditional notions of vendor lock-in and operational control. 80% functional equivalence Open Table Formats as Part of the Modern Data Stack Even before these recent announcements, open table formats had already become integral to modern data lake design. And reciprocally, data lakes have been integral to the modern data stack. A recent by found that 70% of respondents said that more than half of their analytics is or would be in a data lake within three years. This pervasive adoption signifies a paradigm shift in how organizations structure and manage their data, placing a strong emphasis on interoperability, flexibility and performance. survey Dremio It’s no surprise really that cloud-native data lakes and their components and technologies like open table formats have become center stage in the modern data stack. This stands in stark contrast to traditional, monolithic legacy hardware and software sold wholesale to organizations hoping to slap the phrase ‘cloud technology’ onto their aging systems. Becoming cloud-native is more than adding an API – the modern data stack is a modular and specialized ensemble of tools tailored for various data handling facets. It is built for adaptability, born in the  cloud and held to high-performance standards. Features that make the modern data stack a  compelling choice for organizations. The stack's modularity provides a range of options, allowing organizations to craft a bespoke data infrastructure that aligns with their specific needs, fostering agility in the continually evolving data landscape. Despite this continuously evolving range of options, there are defining characteristics that weave through the components of the stack: The modern data stack is designed to seamlessly scale across diverse cloud environments, ensuring compatibility with multiple clouds to prevent vendor lock-in. Cloud-Native: Engineered for efficiency, the stack incorporates components that take a software-first approach and design for performance. Optimized Performance: The stack establishes a standardized communication framework between its components. This promotes interoperability and supports the creation of microservices. RESTful API compatibility: The stack enables independent scaling of computational resources and storage capacity. This approach optimizes cost efficiency and enhances overall performance by allowing each aspect to scale according to specific needs. Disaggregated Storage and Compute: Beyond supporting open table formats, the modern data stack embraces openness in the form of open-source solutions. This commitment eliminates proprietary silos and mitigates vendor lock-in, fostering collaboration, innovation, and improved data accessibility. The dedication to openness reinforces the stack's adaptability across various platforms and tools, ensuring inclusivity. Commitment to Openness: Data Portability and Interoperability as a Business Standard Truly embracing data portability and interoperability means being able to create and access data wherever it is. This approach facilitates flexibility, allowing organizations to harness the capabilities of diverse tools without being constrained by either vendor lock-in or data silos. The goal is to enable universal access to data, promoting a more agile and adaptable data ecosystem within organizations. Understanding that the cloud as an operating model is built on principles of cloud-native technology rather than a specific location is critical to achieving data portability. Some organizations in this endeavor and attempt to buy their way into the cloud at a tremendous cost. The reality is that while cloud adoption presents an opportunity for the average company to , the real impact and true cost savings comes from embracing the cloud operating model on private infrastructure. struggle increase profitability by 20 to 30 percent Many established organizations are actively adopting this philosophy, choosing to repatriate workloads from the cloud and achieving substantial cost savings, with companies like , 37Signals, and a major enterprise security firm from cloud exits. The cloud operating model allows for the coexistence of seemingly contradictory ideas: companies can benefit from migrating to the cloud repatriating workloads. The key determinant is the adoption of the cloud operating model, fundamentally transforming how organizations approach infrastructure, development and technical efficiency. This model optimizes for flexibility, efficiency and long-term success – whether in the public cloud or beyond – and dovetails precisely with the concept of the modern data stack, enabling data portability and interoperability with open table formats. X.com saving an average 60% and Conclusion Recent strides in open table formats by Databricks,  Apache Iceberg and Hudi signify a pivotal moment in data management. Delta Lake 3.0's universal compatibility and expanded support for Apache Iceberg showcase a commitment by both data infrastructure companies and on the ground implementers to seamless data portability and interoperability. These developments align with the inherent modularity of the modern data stack, where open table formats play a central role in achieving performance and compliance standards. This shift is not isolated but intersects with the cloud operating model. Beyond the allure of public clouds, real impact and cost savings emerge by embracing the cloud operating model on private infrastructure. The confluence of open table formats, the modern data stack, and the cloud operating model signifies a transformative era in data management. This approach ensures adaptability across various environments, whether public or private, on-prem on edge. For those navigating data lake architecture complexities, our team at MinIO is ready to assist. Join us at hello@minio.io or on our channel for collaborative discussions as you embark on your data journey. Slack

What the Interoperability Trend in Open Table Formats Signify for Enterprise Data Architectures

এন্টারপ্রাইজ ডেটা আর্কিটেকচারের জন্য ওপেন টেবিল ফরম্যাটে ইন্টারঅপারেবিলিটি ট্রেন্ড কী বোঝায়

Was der Interoperabilitätstrend bei offenen Tabellenformaten für Unternehmensdatenarchitekturen bedeutet

Qué significa la tendencia de interoperabilidad en los formatos de tablas abiertas para las arquitecturas de datos empresariales

Ce que signifie la tendance à l'interopérabilité dans les formats de tables ouvertes pour les architectures de données d'entreprise

ओपन टेबल फॉर्मेट में इंटरऑपरेबिलिटी ट्रेंड एंटरप्राइज डेटा आर्किटेक्चर के लिए क्या दर्शाता है

オープン テーブル フォーマットの相互運用性トレンドがエンタープライズ データ アーキテクチャに何を意味するか

오픈 테이블 형식의 상호 운용성 추세가 엔터프라이즈 데이터 아키텍처에 미치는 영향

O que a tendência de interoperabilidade em formatos de tabelas abertas significa para arquiteturas de dados empresariais

Что означает тенденция совместимости в форматах открытых таблиц для архитектур корпоративных данных

Açık Tablo Formatlarındaki Birlikte Çalışabilirlik Trendi Kurumsal Veri Mimarileri İçin Ne Anlam Taşıyor?

Xu hướng tương tác trong các định dạng bảng mở có ý nghĩa gì đối với Kiến trúc dữ liệu doanh nghiệp

开放表格式的互操作性趋势对企业数据架构意味着什么

Scale Vision Transformers (ViT) Beyond Hugging Face

Scale Vision Transformers (ViT) Más allá de abrazar la cara

हगिंग फेस से परे स्केल विजन ट्रांसफॉर्मर (वीआईटी)

ハグフェイスを超えてビジョントランスフォーマー (ViT) をスケーリング

Scale Vision Transformers (ViT) além de abraçar o rosto

Người vận chuyển tầm nhìn quy mô (ViT) Ngoài khuôn mặt ôm

缩放视觉变形金刚 (ViT) 超越拥抱脸

As I write this on June 29, 2022, the “Data + AI Summit” is on its last day in San Francisco. I’d been thinking about writing about this topic for nearly a year now, but the from to open source all Delta Lake APIs as part of the Delta Lake 2.0 release. They also announced that it will be contributing all enhancements of Delta Lake to The Linux Foundation. That brings us to the “Table Format Wars” subject of this article where we will look at , (created at Uber), and (created at Netflix). For the sake of this article, I’m going to assume you know what the data lakehouse is and how we got here. If you are unfamiliar, this is a to catch you up. announcement Databricks Delta Lake Apache Hudi Apache Iceberg great article Background In the long long ago, in the before time, we had databases, where compute, storage, security, indexing, and all that good stuff was all in one place. As computing needs advanced at a crazy rate, software was developed to address those needs. Now we were dumping structured, semi-structured, and unstructured data (audio, video, images) into storage like AWS S3. We needed a way to understand the structure of what was there. This gave rise to catalogs such as the __ __or AWS Glue that would describe the data. That was a start, but we needed to be able to deal with the data in the files like in a database and be able to insert/delete/update records, and that gave rise to the Hive table format. It was quite primitive compared to the options available today, and if I recall correctly, didn’t work with metastore yet, so users had to know the layouts of the files. Hive Metastore Today Databricks built and released Deltlake as a table format in April 2019, but the open-source version was pretty limited and Databricks was really the only company doing any serious work on it. That’s what made their announcement about open-sourcing a bunch of stuff so interesting. I read it as a bit of a panic move as Iceberg is getting so much vendor adoption and so many vendor contributors. Iceberg is also the newer kid on the block open-sourced in May 2020. Hudi started life at Uber in 2016 and was open-sourced in 2017, and is used at some dang big companies other than Uber, like the Robinhood trading platform, Amazon, Bytedance, creators of TikTok, and many others. Features The way these table formats work to handle upserts, deletes, etc., is generally one of two methods, they are: Copy on Write (CoW) Merge on Read (MoR) Keep in mind that these changes to the files are coming in as change logs, which means the latest versions need to be dealt with for queries, or time travel. Time travel is a cool feature of this configuration, but I’m not going to address it in this article. MoR tends to be faster than CoW, but this is a pretty detailed topic that you should research in-depth. Let’s do a high-level comparison of the three table formats to get you started. For the below grid, I’m borrowing some research from a webinar. Dremio Feature Overview Delta Lake Hudi Iceberg ACID Transactions Yes Yes Yes Partition Evolution No No Yes Schema Evolution Partial Partial Yes Time Travel Yes Yes Yes File Formats Supported Parquet Orc, Parquet Avro, Orc, Parquet Schema Evolution Add Column Yes Yes Yes Drop Column No Yes w/Spark Yes Rename Column Yes Yes w/Spark Yes Update Column Yes Yes w/Spark Yes Reorder Column Yes Yes w/Spark Yes Change partitioning w/out rewriting table No No Yes Use transforms of columns to specify partitions Partial No Yes Require understanding of table partitioning Yes No Yes File Pruning Yes Yes Yes Read Support Athena Yes Yes Yes Beam Yes No No BigQuery No Yes No Databricks SQL Yes No No Drill No No Yes Flink Yes Yes Yes Hive Yes Yes Yes Impala No Yes Yes Presto Yes Yes Yes Redshift Yes Yes No Snowflake Yes No Yes Sonar Yes No Yes Spark Yes Yes Yes Trino Yes Yes Yes Write Support Athena No No Yes Flink Yes Yes Yes Impala No Yes Yes Databricks Photon Yes No No Presto No No Yes Sonar No No Yes Spark Yes Yes Yes Trino Yes No Yes Observations Part of the reason this ecosystem evolved was around performance, but part of it had to do with how cloud providers charge for their platforms. It is much cheaper for storage than it is for compute, so if you can just query your raw storage without putting it into a conventional database, you’ll reduce your costs. By NOT putting it in a database of some sort, we’ve had to develop a variety of file formats like Parquet, catalog systems to describe a variety of file formats so they present as a schema, query engines, security plug-ins, table formats to deal with transactions, indexing, and more. The appeal of systems like Redshift, Snowflake, Yugabyte, CockroachDB, and others, is that all those things tend to be built in, just like the databases of old. Yes, there is a lot of flexibility in this scenario as you can use all the bits and pieces that best suit your situation, but imagine if Amazon were to suddenly change their pricing policies with Graviton 5 (I’m just throwing out an idea here) because they got compute so cheap that they decided to make compute cheaper than storage and really drop egress fees. You could see a massive collapse of a segment of the tech sector. Kind of a scary thought. Summary DataBeans published a comparing Delta, Hudi, and Iceberg where Delta comes out on top, the day before I was starting this article. The next day, Onehouse, a commercialization of Hudi by the Hudi developers as I understand it, published a between Delta and Hudi. Both provide details and access to the files so you can try them yourself. I think Onehouse makes some good arguments and it is odd that the DataBeans article is not attributed to any writers, and it came out at the same time as the Delta announcement. benchmark rebuttal benchmark The tests performed were ones that would show Delta in a better light with default configurations. It all seems just a little too convenient, meaning, it seems like it was more of a paid placement than an organic comparison, but I could be wrong (this article is not paid for, I did it on my own time and dime). While Delta and Hudi have broad consumer adoption, the former because it is part of the Databricks product and the latter because of being first to market, I think Iceberg is likely going to be the eventual winner in this space based on the commercial support that I’m seeing, but you never know how the market can change from some unexpected innovation. As to what is best for your environment, that’s going to be up to your specific needs, I’m simply talking about the tech adoption. The nice thing about open-source is that it isn’t company-dependent, so you can keep using that tech regardless of if the commercial company that sold you support continues to exist or not.

Stories, News & Mentions about DATABRICKS

Databricks’s Stories on HackerNoon

Databricks's News & Mentions