Highlights from the Data and AI Summit (former Apache Spark Summit) for the Busy IT Professional Right now, I am actually waiting for the keynote of day 2 to kick off. So let me summarize the announcements from day 1 of the Data and AI Summit (DAIS) for you. Delta.io Delta Lake has grown up :-). The Linux Foundation-owned, open-source project that brings reliability to your data lake matured to version 1.0 now. Happy Birthday, Delta! Key features of Delta 1.0: Delta.io , JVM implementation that understands tx protocol, but does not need SparkRust implementation for Delta 1.0, also Support for Ruby and Go. Delta Standalone Reader Delta 1.0 supports Spark 3.1 - also improves predicate pushdown. Delta Sharing A brand new open source project for data sharing was announced at DAIS. With you can share massive, live data from your Lakehouse between clouds or on-premises. It’s secure, fast, cheap, and reliable using underlying cloud storage systems such as S3, ADLS, and GCS. And it’s open source & open standard. Recipients accessing shared data can connect to it through pandas, Tableau, or dozens of other systems that implement the open protocol. . Delta Sharing directly #DataAISummit Unity Catalog Databricks rolled out a data catalog, . Unity Catalog solves a major issue. Imagine a CSV file stored in your S3 data lake and you want to grant access to certain rows only? Unity Catalog Unity Catalog enforces permissions at the row, column, or view level instead of the file level. It governs tables and ML models. Simply use ANSI SQL standard GRANT statements, or discover data assets from the UI. Works for the Lakehouse on all clouds. Delta Live Tables My fav announcement: Delta Live Tables. Data flow made simple: Specify the outcomes that a pipeline needs to achieve using SQL or Python. Treat your transformations and data quality expectations as code. #DataAISummit And now for day 2 of DAIS! Databricks Machine Learning Databricks Machine Learning brings together managed , and introduces new components, such as AutoML and the Feature Store, and supports the full ML lifecycle. MLflow Feature Store MLflow integration enables the to package up feature lookup logic hermetically with the model artifact. When an MLflow model that was trained on data from the Feature Store is deployed, the model itself will look up features from the appropriate online store. Feature Store The Databricks Feature Store automatically tracks the data sources used for feature computation, as well as the exact version of the code that was used. AutoML allows you to quickly build and deploy machine learning models by automating the heavy lifting of , and AutoML detects the best preprocessing, ML model, and hyperparameters for you and creates a notebook with all steps required. It automatically tracks trial run metrics and parameters with MLflow and easily enables teams to register and version control their models in the for deployment. You define the max runtime that you want to spend to solve the task. AutoML preprocessing feature engineering, model training/tuning. Databricks Model Registry There are more video resources from DAIS 2021 that I recommend: The , , , , , and . fireside chat with Bill Inmon (DWH inventor) Delta 1.0 announcement Delta Sharing SQL and Photon updates Unity Catalog Delta Live Tables Slice & DAIS 2021 — EMEA Live Event Join us for the first that talks about all the new announcements in a beginner-friendly way. session Slice & DAIS Previously read behind a paywall at https://medium.com/geekculture/what-is-new-in-data-and-ai-2021-376cefe67fb3

Flow

Apache

Share Large Amounts of Live Data With Delta Sharing and Docker

Read My Stories

Nominated for 2022 - HackerNoon Contributor of the Year - Open Source

Too Long; Didn't Read

Make resilience your competitive advantage

Top 7 Announcements from Data and AI Summit 2021

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Share Large Amounts of Live Data With Delta Sharing and Docker

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

Share Large Amounts of Live Data With Delta Sharing and Docker

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps