GlareDB is a fast SQL database for querying and analyzing distributed data. GitHub: https://github.com/GlareDB/glaredb

GlareDB

Introduction It has been a while since my last “What the heck is?” article, and I’ve recently seen some rapid growth from and wanted to learn more. What really piqued my interest was the recent announcements of support for and a new model. So, what the heck is GlareDB? Let’s take a look! GlareDB Apache Iceberg hybrid execution Overview GlareDB is an project utilizing the project, part of the Apache Arrow project. DataFusion is a fast, extensible query engine for building high-quality data-centric systems in , using the in-memory format. It offers SQL and Dataframe APIs and built-in support for CSV, Parquet, JSON, and Avro. There are also as well as extensive customization possibilities. GlareDB is adding many features on top of it, such as cloud storage and the aforementioned hybrid execution feature, providing a  layer on top of various compute engines that can: open-source DataFusion Rust Apache Arrow Python bindings Query local and remote files Query other databases and data sources Store data and queries (as views) Copy data from sources to destinations Interop with DataFrame libraries in Python Run one-off queries from the command line They describe how it fits in the stack in this diagram: It supports data located on GCS or S3 of the following types: BigQuery MongoDB (early release) MySQL Postgres Snowflake Preliminary Iceberg support Redshift (coming soon) ClickHouse (coming soon) They are quickly adding support for various engines, so this list could be incomplete by the time you read this. What can I do with it? At first blush, you look at this and think, hey, this seems a lot like in that it is a federated query engine. On second glance, it seems kind of like for a couple of reasons. The first is that, like DuckDB, GlareDB is a single, tight executable but written in Rust instead of C++. Second, they also support having this model (MotherDuck did it first), which I’ll cover shortly. Trino Motherduck hybrid execution Given that Trino is written in Java, that means there is a lot of Java ecosystem you need to deal with if you want to use it. Sure, there are pre-built Docker containers around that can shorten this path, but generally, if you are “just trying to do something,” then you have a heavy lift to install and set up Trino. With GlareDB, you have a single executable to download and use or make use of their SaaS product, which looks like this when you first use it: Now to Hybrid Execution. I’ll paraphrase some of what GlareDB had to say in their blog post on the topic. Say you have a CSV list of user IDs that had gotten extracted from some other tool from your database. Now, you want to enrich that data with some of the user's demographic information from your database. We’ll say our table name is user_demo and our CSV file is user_id.csv, and our query would look something like this: SELECT
   m.user_id,
   m.first_name,
   m.last_name,
   m.birth_date
FROM
   user_demo m
INNER JOIN '/user_id.csv' u on m.user_id = u.id
GROUP BY m.user_id; Clearly, this is a simple example, but you could enhance it to get information out of other joined tables as well. You can also go in the other direction, where you have some local file with a key field and some data you are interested in that you can join to a table in a database where that extra data in the file doesn’t exist in the database. This has the advantage of not having to go through the process of creating a new table and loading it for this ad-hoc report, thus saving a lot of time. That’s all just meant to give you a quick tickle about what GlareDB can do and where it is at currently. The docs and blogs on their site are well done, making it pretty quick to jump in. Summary GlareDB is very interesting, and I appreciate how quickly they are iterating and updating the software. I need to spend some more time thinking about how it plays in the , or space. Between the speed and the federated queries, there are some exciting possibilities. I really like the new hybrid execution, which could shortcut work in various situations. Try out a free account yourself if you’d like to give it a spin at . Trino StarRocks, DuckDB GlareDB You can read the other “What the heck” articles at these links: (I was pretty out front on this one.) What The Heck Is DuckDB? (I was out front on this one, too.) What the Heck Is Malloy? (slower, but also growing) What the Heck is PRQL?

What the Heck is GlareDB?

Introduction Late in 2023, I was considering writing an article about Acorn Labs' work simplifying Kubernetes containers. I even started it and wrote a few paragraphs, but then I got busy with other things. At the time, I was going to revisit it in February 2024, they made a couple of announcements; one was for Rubra, and the other was more of a personal project by one of the co-founders, Darren Shepherd, called GPTScript. This project was so exciting that a month later, Acorn announced they were changing their company focus to GPTScript (not sure what would happen to Rubra). I’ve been following all the developments on this project, closer than I typically do, because I find it absolutely amazing as enablement technology. It reminds me of the philosophy behind a product I worked with back in the 80s and 90s called MPEX from VESOFT, written by Eugen Volokh. What Darren has done with GPTScript is kind of hard to wrap your head around at first, especially for a programmer. I haven’t spent enough time with it yet to fully understand all the implications, but I have a lot of thoughts about it. In this article, I’ll walk through some basic introductions and examples and give you some thoughts about where you could take it. Getting Started I worked off the docs and various videos and examples; I gave this a go on my Windows 11 laptop. Let’s get this sucker installed: winget install gptscript-ai.gptscript
$env:OPENAI_API_KEY = 'your-api-key' The docs suggest running a “hello world” example at this point, but that isn’t very interesting, so let’s try the “bob” example. Also, note that there are many built-in examples you can get a list of by issuing the gptscript—-list-tools command and an online ‘examples’ directory. What does “bob.gpt” look like? This presents us with some obvious questions, like what this all means and how a value of ‘How are you doing?’ ended up in the ‘question’ argument. GPTScript operates around the notion of tools where a tool is described in a GPT file. Each file can be made up of more than one named tool, separated by three dashes. Borrowing from the docs, Tool parameters are key-value pairs defined at the beginning of a tool block before any instructional text. They are specified in the format key: value. The parser recognizes the following keys (case-insensitive and spaces are ignored): A tool starts with a preamble that defines the tool's name, description, args, available tools, and additional parameters. The preamble is followed by the tool's body, which contains the instructions for the tool. Comments in the preamble are lines starting with # and are ignored by the parser. Comments are not really encouraged as the text is typically more useful in the description, argument descriptions, or instructions. I’m still not sure why Bob was giving a default value for the argument. It would change if I gave it an argument, so that was a bit of a mystery. I decided to try the samples-readme.gpt script, which looks like this: Reading this, I understood that it would give me a summary of each gpt file in the examples directory. When I first tried it without an argument, it failed to find any input, so I tried with an example file as an argument, and it worked. Then I tried it with their larger example script for hacker-news-headlines, and instead, it did what I thought it should have done the first time. It created a readme.md file in the examples directory with a summary of each example. I’ve seen people in the community starting to generate some very cool tools that are being added to the GPTScript tools web page. I had a little bit of mixed results with my tests. Most things worked as expected, but I had a few things that didn’t, and I didn’t have the opportunity to dig deeper. That said, this is clearly powerful, and it is just scratching the surface. Summary As a lifelong programmer, this feels weird; it involves doing things in ways that seem like something is missing. This makes GPTScript fascinating and scary as if black magic is involved somehow. The natural language and loose feel of it take a little getting used to, but as you think about it, you start to realize how amazing this can be as an enablement technology. One of the first things that came to mind, but I didn’t test, was could this be used to drive OpenFoam? OpenFoam is open-source CFD software that is rather notorious for how difficult it is to use, which is why I never learned how to use it. What if I could just describe what I wanted to do with it, and it worked? Imagine doing this with any software with an API you could talk to. My mind starts to explode thinking about it. Acorn has created an online “tools” catalog that you can utilize, and they are iterating at an insane pace. This is really exciting. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB?
What the Heck Is Malloy?
What the Heck is PRQL?
What the Heck is GlareDB?
What the Heck is SeaTunnel?
What the Heck is LanceDB?
What the heck is SDF?
What the Heck is Paimon?
What the Heck is Proton?
What the Heck is PuppyGraph? Introduction Introduction Late in 2023, I was considering writing an article about Acorn Labs ' work simplifying Kubernetes containers. I even started it and wrote a few paragraphs, but then I got busy with other things. At the time, I was going to revisit it in February 2024, they made a couple of announcements; one was for Rubra, and the other was more of a personal project by one of the co-founders, Darren Shepherd , called GPTScript . Acorn Labs Acorn Labs Darren Shepherd Darren Shepherd GPTScript GPTScript This project was so exciting that a month later, Acorn announced they were changing their company focus to GPTScript (not sure what would happen to Rubra). announced announced I’ve been following all the developments on this project, closer than I typically do, because I find it absolutely amazing as enablement technology. It reminds me of the philosophy behind a product I worked with back in the 80s and 90s called MPEX from VESOFT, written by Eugen Volokh. What Darren has done with GPTScript is kind of hard to wrap your head around at first, especially for a programmer. I haven’t spent enough time with it yet to fully understand all the implications, but I have a lot of thoughts about it. In this article, I’ll walk through some basic introductions and examples and give you some thoughts about where you could take it. Getting Started I worked off the docs and various videos and examples; I gave this a go on my Windows 11 laptop. Let’s get this sucker installed: docs docs winget install gptscript-ai.gptscript
$env:OPENAI_API_KEY = 'your-api-key' winget install gptscript-ai.gptscript
$env:OPENAI_API_KEY = 'your-api-key' The docs suggest running a “hello world” example at this point, but that isn’t very interesting, so let’s try the “bob” example. Also, note that there are many built-in examples you can get a list of by issuing the gptscript—-list-tools command and an online ‘ examples ’ directory. examples examples What does “bob.gpt” look like? This presents us with some obvious questions, like what this all means and how a value of ‘How are you doing?’ ended up in the ‘question’ argument. GPTScript operates around the notion of tools where a tool is described in a GPT file. Each file can be made up of more than one named tool, separated by three dashes. tools Borrowing from the docs, Tool parameters are key-value pairs defined at the beginning of a tool block before any instructional text. They are specified in the format key: value. The parser recognizes the following keys (case-insensitive and spaces are ignored): Tool parameters are key-value pairs defined at the beginning of a tool block before any instructional text. They are specified in the format key: value. The parser recognizes the following keys (case-insensitive and spaces are ignored): A tool starts with a preamble that defines the tool's name, description, args, available tools, and additional parameters. The preamble is followed by the tool's body, which contains the instructions for the tool. Comments in the preamble are lines starting with # and are ignored by the parser. Comments are not really encouraged as the text is typically more useful in the description, argument descriptions, or instructions. A tool starts with a preamble that defines the tool's name, description, args, available tools, and additional parameters. The preamble is followed by the tool's body, which contains the instructions for the tool. Comments in the preamble are lines starting with # and are ignored by the parser. Comments are not really encouraged as the text is typically more useful in the description, argument descriptions, or instructions. I’m still not sure why Bob was giving a default value for the argument. It would change if I gave it an argument, so that was a bit of a mystery. I decided to try the samples-readme.gpt script, which looks like this: Bob samples-readme.gpt Reading this, I understood that it would give me a summary of each gpt file in the examples directory. When I first tried it without an argument, it failed to find any input, so I tried with an example file as an argument, and it worked. Then I tried it with their larger example script for hacker-news-headlines , and instead, it did what I thought it should have done the first time. It created a readme.md file in the examples directory with a summary of each example. hacker-news-headlines readme.md I’ve seen people in the community starting to generate some very cool tools that are being added to the GPTScript tools web page. I had a little bit of mixed results with my tests. Most things worked as expected, but I had a few things that didn’t, and I didn’t have the opportunity to dig deeper. That said, this is clearly powerful, and it is just scratching the surface. Summary As a lifelong programmer, this feels weird; it involves doing things in ways that seem like something is missing. This makes GPTScript fascinating and scary as if black magic is involved somehow. The natural language and loose feel of it take a little getting used to, but as you think about it, you start to realize how amazing this can be as an enablement technology. One of the first things that came to mind, but I didn’t test, was could this be used to drive OpenFoam ? OpenFoam is open-source CFD software that is rather notorious for how difficult it is to use, which is why I never learned how to use it. What if I could just describe what I wanted to do with it, and it worked? Imagine doing this with any software with an API you could talk to. OpenFoam OpenFoam My mind starts to explode thinking about it. Acorn has created an online “ tools ” catalog that you can utilize, and they are iterating at an insane pace. This is really exciting. tools tools Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the heck is SDF? What the Heck is Paimon? What the Heck is Proton? What the Heck is PuppyGraph? What The Heck Is DuckDB? What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is SeaTunnel? What the Heck is LanceDB? What the Heck is LanceDB? What the heck is SDF? What the heck is SDF? What the Heck is Paimon? What the Heck is Paimon? What the Heck is Proton? What the Heck is Proton? What the Heck is PuppyGraph? What the Heck is PuppyGraph?

What the Heck Is GPTScript?

What the heck is ? That was the first thing I asked myself when I came across it in the Summer of 2023. It was my involvement with that brought it to my attention; they wanted to add Iceberg support to PuppyGraph, but just what the heck is it? PuppyGraph Apache Iceberg This blog is going to be more of a hot take on PuppyGraph to get you thinking about how you might use it in your own projects. I have no affiliation with the company or project other than thinking it was pretty cool. Co-founder recently (Feb 2024) gave a presentation at the that was interesting, and well received, according to my friends that were there. Weimo Liu Chill Data Summit What is PuppyGraph? Simply, PuppyGraph is a cloud-native graph data lakehouse providing a graph analytics engine for your data. They address graph scalability through the auto-sharding of data so the compute and storage are separate, much like the lakehouse design. So, they provide a graph data warehouse, data lake, and multi-data models on a single copy of your data. That means you can do some pretty cool graphing on your data in one of the supported formats. What can it connect to? PuppyGraph has rapidly added support for various platforms, catalogs, and connection engines. Currently, we see: Apache Iceberg Apache Hudi Delta Lake MySQL PostgreSQL DuckDB BigQuery Redshift LanceDB (coming soon) JDBC Catalog Data Lake Catalog Hive Metastore AWS Glue Their SaaS interface also gives you direct access to both a Gremlin and Cypher console to perform graph queries, in addition to a graph notebook, which uses Jupyter. Using PuppyGraph A Docker container is provided to allow you to get started on a local machine. You’ll need a schema defined in JSON format that will define your data layout to PuppyGraph. Once you ingest that and it is verified, then away you go. The integrated graph browser is pretty nifty. You can easily zoom in/out to see the clustering and attributes in addition to queries. Zooming in further, we can see more of the details: Clicking on a node will give us a pop-up of details: This allows you to explore different vertices and edges easily. These static pictures don’t really represent how fast the performance is or how much fun it is to bounce around your data. I should have utilized some genealogical data for fun. Because they are using the Gremlin and Cypher query languages, that means any 3rd party UI tool will also be compatible. A real advantage here is that PuppyGraph works on the data where it lives and isn’t making you copy it elsewhere. Without going into the particulars on a specific platform, this gives you a general idea of what features and functions are available. Summary Certainly, graph databases and their representation don’t apply generically as a structured database does, but we are seeing more and more how these kinds of data representations are being used to model the real world. I didn’t see that this is an open-source project, and I didn’t find it on GitHub. There is no mention of pricing, so I’m not sure where they are going with all of this. The documentation isn’t amazing, but it seems to be enough to get started and try it out. Overall, this is a fun project to play with. I need to percolate on it more to see where I might use it, but I can envision some interesting use cases combining it with other self-contained projects like DuckDB and LanceDB. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the heck is SDF? What the Heck is Paimon? What the Heck is Proton?

What the Heck is PuppyGraph?

This series of articles has been a lot of fun for me as I have learned about and explored new technology. It’s also fun to see what has been catching on since I first discovered it. My last article on was incredibly popular, much to my surprise, but it seems I wasn’t the only one interested in what the heck it was. Thanks to that article, I ran across the open-source Apache 2.0 licensed project, , sponsored by . It’s a SQL database that accommodates both historical and streaming data. Written in C++ and powered by , the focus is on simplicity and performance. With a single executable, installation is simple. Apache Paimon Proton Timeplus ClickHouse A trend I’ve seen is that more and more real-time analytics applications are being built, but you don’t want to build them twice. Once for streaming and once for the historical backfill. There would be definite advantages to having a single platform that could query in either batch or streaming mode or even a hybrid mode where you are joining historical data to a stream of incoming data. It appears that Proton was built to do just that. Proton Overview In a nutshell, we have a ClickHouse database, and Timeplus has added support for streaming services. That should get you a Flink-like query engine and Kafka-like streaming storage with that ClickHouse database. So, what does that look like? The dotted line is where Proton comes in. I suggest reading through the docs to get a good sense of what is possible. architecture To create a random stream of data and query it with Proton, we can do something like this: -- Create a stream with random data.
CREATE RANDOM STREAM devices(device string default 'device'||to_string(rand()%4), temperature float default rand()%1000/10);

-- Run the long-running stream query.
SELECT device, count(*), min(temperature), max(temperature) FROM devices GROUP BY device;

┌─device──┬─count()─┬─min(temperature)─┬─max(temperature)─┐
│ device0 │    2256 │                0 │             99.6 │
│ device1 │    2260 │              0.1 │             99.7 │
│ device3 │    2259 │              0.3 │             99.9 │
│ device2 │    2225 │              0.2 │             99.8 │
└─────────┴─────────┴──────────────────┴──────────────────┘ Proton Features Proton has many nifty features; one that struck me immediately was the ability to create a materialized view to save specific events in Proton. Borrowing from the documentation, let’s say you have a Kafka stream reporting web events, and you want to save the broken link reports so you can query them later, even with Kafka down or the events removed. It would look something like this: create materialized view mv_broken_links as
select raw:requestedUrl as url,raw:method as method, raw:ipAddress as ip, 
       raw:response.statusCode as statusCode, domain(raw:headers.referrer)
as referrer
from frontend_events where raw:response.statusCode<>'200'; Then, if you want to directly query the materialized view and make a bar chart from the data, it would look like this: -- streaming query
select * from mv_broken_links;

-- historical query
select method, count() as cnt, bar(cnt,0,40,5) as bar from table(mv_broken_links) 
group by method order by cnt desc;

┌─method─┬─cnt─┬─bar─┐
│ GET    │  25 │ ███ │
│ DELETE │  20 │ ██▌ │
│ HEAD   │  17 │ ██  │
│ POST   │  17 │ ██  │
│ PUT    │  17 │ ██  │
│ PATCH  │  17 │ ██  │
└────────┴─────┴─────┘ Some of this functionality reminds me of , a company I worked at a few years ago. Upsolver Drivers for other languages are available for Java, Go, and Python. Using Proton with something like would be a minimal footprint for streaming historical data. Redpanda There are a lot of other features available, but this isn’t meant to be a tutorial. I want to do a light explanation and draw attention to some features. The docs are concise and, overall, well written, certainly better than many open-source projects. Summary While I don’t personally need this kind of arrangement at the moment, I’ve certainly been at places and seen companies where this would be very, very cool to have. As cool as this guy? Probably not, but then again, nothing is :). Frivolity aside, the Proton team has done an excellent job documenting the project and making it as simple to install and use as possible. I love these single-binary projects that don’t need a vast Java ecosystem with tons of dependencies. Make no mistake, though, Timeplus has a commercial version that gives you more capability than the stock Proton release. However, they seem to be very supportive of Proton and welcoming of the community. Check out my other What the Heck is… articles at the links below: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the heck is SDF? What the Heck is Paimon?

What the Heck is Proton?

হেক প্রোটন কি?

Was zum Teufel ist Proton?

¿Qué diablos es el protón?

Qu'est-ce que c'est que Proton ?

आख़िर प्रोटोन क्या है?

プロトンって一体何なの？

양성자는 도대체 무엇입니까?

O que diabos é próton?

Что, черт возьми, такое Протон?

Proton nedir?

Proton là cái quái gì vậy?

质子到底是什么？

Introduction You’ve heard of data warehouses, you’ve probably heard of data lakes and the data lakehouse, but have you heard of the “Streamhouse”? Well, that’s what “they” call the new data tech unlock possible with . So, just what the heck is it? Let’s find out! Apache Paimon A company named Ververica, the original creator of , looks pretty excited about Paimon, but Jingsong Lee primarily created it at Alibaba. The purpose initially was to address the inability to query a Flink Dynamic Table, which resulted in the community proposing . Apache Flink FLIP-188: Introduce Built-in Dynamic Table Storage Paimon can be thought of as a table format for batch and stream processing. I’ve written a couple of times about the big three table formats and . here here Paimon Overview The short version of what Paimon does is that it provides a streaming storage layer and extends Flink's capability to do stream processing directly on the data lake. The use case seems to be tightly coupled to Flink at the moment by giving you a more flexible and persistent storage layer for your streaming data. Instead of using the ‘InMemory Catalog’, you would set up a ‘Paimon Catalog’ like other table formats. That would look something like this: CREATE CATALOG paimon WITH (
   'type' = 'paimon',
   'Warehouse' = '<path to your warehouse>'
);
USE CATALOG PAIMON; To explain the flow, I’m going to borrow a screenshot from the by Giannis Polyzos, excellent blog If you aren’t familiar with what an LSM Tree (Log-Structured Merge) is, then I suggest by Vishal Rana. I think he did a nice job explaining it. this blog So, Paimon can be used to replace message queues, providing a storage layer for Flink that is a table format. As a streaming data lake platform, it allows users to process data in both batch and streaming modes, supporting: High-speed data ingestion Change data tracking Real-time analytics High throughput data writing Low-latency data queries Batch writes and reads Streaming updates Changelog producing Where It Applies The first few use cases that occurred to me were gaming, IoT devices, stock trading, and services like Uber. There is generally a lot of data coming in quickly, and you probably need to be able to do something actionable from it. I remember having to deal with combining streaming data from ad impressions and ad clicks to get insights on advertising data as another example. Having this kind of a materialized view and a table format defined on it would certainly have been helpful. I bet you can think of some use cases that would help you out. Borrowing another screenshot from Giannis Polyzos, we see how we are getting to what is being called a “Streamhouse.” I found this very interesting after my time in the (compute engine) and (table format) communities. PrestoDB Apache Iceberg Streaming would come up fairly often with Iceberg, and there is an argument out there that the Apache Hudi table format is better for that use case. Paimon can also have some issues with Change Data Capture (CDC) ingestion into a data lake. It simplifies the CDC pipeline, supporting synchronizing CDC data with schema changes, streaming changelog tracking, and a partial-update merge engine. Summary As I researched for this blog, I thought Paimon and Seatunnel would make an exciting combination for a real-time data warehouse. Then I ran across this blog, “ ” from back in May 2023, so clearly, I’m not the first to think about this idea. Apache SeaTunnel and Paimon: Unleashing the Potential of Real-Time Data Warehousing With these new projects, we’re seeing an interesting shift in the streaming and real-time space, and I’m here for it. I wish I personally had a use case to try these things out on a big, live system. We’re seeing more and more clever technology being developed to address new problems that arise. I wonder how the Amazon S3 Express One Zone Storage Class announcement might impact the “real-time” space. It’s all very exciting to watch. You can read more "What the heck" articles at the following links: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB? What the Heck is SDF?

What the Heck Is Apache Paimon?

Introduction The year 2023 has seen significant developments in innovation, adoption, and competition. HashiCorp ignited a substantial controversy when it altered the licenses of its products to be less open-source-friendly. This quickly led to the emergence of Terraform forks, such as OpenTofu. We also observed increased competition among established players like Fivetran and dbt, often collaborating to provide solutions. This brings us to the topic of this blog post—the Rust-based alternative to dbt, known as SDF (Semantic Data Fabric). So, what exactly is SDF? SDF Overview The origins of SDF can be traced back to Meta. It is written in Rust, meaning a single compact binary with no packages to manage. Once downloaded, you're ready to get started. SDF is built on Datafusion, which is part of the Apache Arrow project. With SDF, you'll primarily be working with YAML and SQL—languages that should be familiar to any data engineer. At its core, SDF is a compiler and build system that applies static analysis to SQL code. It can then generate a robust dependency graph for a clearer view of your data assets. This feature enables easier detection of potential problems and optimization issues. One of SDF's unique features is the capacity to annotate your SQL sources with metadata. This can range from simple types and classifiers (like PII) to table visibility and privacy policies. SDF refers to these as Checks. Here are some examples of the types of Checks you can perform: Data Privacy Check: Ensures all personally identifiable information (PII) is suitably anonymized. Data Ownership Check: Confirms that every table has an owner—a requirement of GDPR. Data Quality Check: Prevents calculations that mix different currency types (for example, preventing £ + $ ) The resulting information schema extends beyond standard features. You can use SQL to explore the information schema and even write your own Checks to integrate into your workflows. Additional Features The SDF engine includes a multi-dialect SQL compiler, a static analyzer, a dependency manager, and a build cache. The following images illustrate the , , and commands. deploy describe lineage These features are available in their CLI, and a cloud version offers an interactive interface to explore the catalog and lineage. You can easily zoom in and out, click on a node, and delve deeper. Here is a zoomed-out image: As previously mentioned, the metadata and configurations for SDF are in YAML and form an SDF Workspace. SDF extends YAML's typical configuration usage by allowing you to specify data asset definitions simultaneously, referred to as definition blocks. These blocks can be used to define or enrich tables, functions, classifiers, etc. Borrowing an example from their documentation, here's an illustrative SDF YAML workspace. These classifiers can conveniently tag your columns and their descendants throughout a project. Coupled with available functions, you gain substantial control and visibility over your data models. The compilation step and helpful error messages contribute to a comprehensive environment. Summary This project's compact nature is impressive; you can install it locally and experiment with minimal effort. It proposes a different approach to dbt, which may cause resistance for some. However, uncertainty remains about how this would perform in extensive, complex environments where there's a desire to deploy thousands of modules. Nevertheless, the Checks feature could be the solution to this concern. SDF is undeniably a valuable project that can meet many use cases. If you're weary of dbt or considering something akin to dbt, then SDF should probably be on your shortlist. You can read more "What the heck" articles at the following links: What The Heck Is DuckDB? What the Heck Is Malloy? What the Heck is PRQL? What the Heck is GlareDB? What the Heck is SeaTunnel? What the Heck is LanceDB?

What the Heck Is SDF?

Introduction I started seeing early in 2023, and my first thought was that it might be an attractive fit in the ecosystem or as a general replacement for the columnar file format. I was thinking too small, though. LanceDB Apache Iceberg Parquet It is an in-process multi-model serverless vector database written in Rust that is cloud-native and open-source. It’s not to be confused with the Lance columnar format (which I was doing), which is also written in Rust and a more appropriate comparison to Parquet but is fundamental to LanceDB. It is also compatible. Apache Arrow So, what we have is a SQL-compatible vector database that supports vectors, images, text, and videos with full-text search. It is also said to be very fast, but you’ll need to do your own tests to see how it does on your own stack. Why a Vector Database? Two letters, AI. A vector database is at the core of the data repository used to train Large Language Models (LLM). LanceDB has gone the extra mile to provide a GitHub repository with a few vector recipes that you can find . here So, you should store embeddings from a machine learning model, for example, to search for images using written descriptions. The challenge here is the “ ”; you can be fast or accurate but can’t be both; you have to pick one. LanceDB first went for speed, then did a lot of tuning to improve accuracy. Curse of dimensionality This leads to “Embeddings,” which are high-dimensional floating-point vector representations of a query or the data. You can embed anything using an appropriate embedding model or function. The position of the embedding in a vector space has semantic significance depending on the type of modal and training you are using. LanceDB supports “explicit” and “implicit” data vectorization methods. At this stage, we’re getting into some deep water concerning how this all works, and it is beyond the scope of what I’m trying to convey in this blog. I’ll share an image from the LanceDB docs that illustrates how similar entries cluster within a vector system. The LanceDB ecosystem provides all the latest and most commonly used tools for this space to make it as convenient as possible to get started. You can use Python and Javascript to process your data into LanceDB, using popular Python machine learning and columnar data packages. A native Typescript SDK is available to allow for vectorless search as part of serverless functions. What isn’t listed here is the new integrations, making that LLM work even easier. LangChain That’s not all though, so to put it in a list, we also see: Duckdb LangChain JS/TS LlamaIndex Voxel51 There is a great, recent blog, “ ” by Ayush Chaurasia, that I highly recommend. He gives a quick walkthrough using Nextjs, LanceDB, and Roboflow’s CLIP inference API. If you'd like to dive a little deeper, it’s a specific deep dive with example code. Serverless Multi-Modal search application Summary Obviously, LanceDB isn’t a general-purpose database, it has a very specific use case, and from what I can tell, it’s a solid solution for that use case. It is an extremely fast vector database that can be used specifically in AI applications. I can envision some really useful scenarios for building up your own LLM around product blogs and documentation with this, to enable users to write specific questions and get more tailored responses than trying to read through tons of blogs and docs. This could be the beginning of a tide shift in how we provide information to the public. Finally, a LanceDB Cloud is coming, and at the time of this writing, in October 2023, you can sign up to be notified about it at this . I’m excited to give that a try and run some tests on ideas that have been bouncing around in my head since ChatGPT started grabbing everyone's attention. link You can read the other “What the heck” articles at these links: (I was pretty out front on this one.) What The Heck Is DuckDB? (I was out front on this one, too.) What the Heck Is Malloy? (slower, but also growing) What the Heck is PRQL? (growing quickly) What the Heck is GlareDB? (interest is hot) What the Heck is SeaTunnel?

What the Heck Is LanceDB?

I started seeing chatter about in early 2023 and was low-key keeping an eye on it. The project started in 2017 as Waterdrop and grew out of the contributions from the creator of which supports SeaTunnel as a task plugin. Apache SeaTunnel Apache Dolphin, I had some initial issues getting my head around what SeaTunnel is and why I should care about it. That means I will keep this relatively high level to at least answer those questions. With that, let’s jump in. What is SeaTunnel? They describe it as “a high-performance, distributed, massive data integration tool that provides an all-in-one solution for heterogeneous data integration and data synchronization.”  It comprises three main components: Source connectors Transform connectors Sink connectors. Many source connectors are available; as of the current version, 2.3.3, a list is available . It includes formats such as relational, , and graph, as well as distributed file systems like HDFS and object stores like S3. here NoSQL A transform connector comes into play if the schema between your source and sink differ, essentially mapping your data. The sink is the other side of the source, but now you are writing instead of reading. A complete list of sink connectors as of version 2.3.3 is available . As of this writing, it is claimed that over 100 connectors currently exist. here With these components, SeaTunnel can solve common problems found with data integration and synchronization. So, it provides high-performance data synchronization for real-time and batch data. The poorly translated claim in the docs is that it can “synchronize hundreds of billions of data per day in real-time.” I’m not sure what that claim is, but it's probably pretty fast, considering that companies like Alibaba use it. Apache SeaTunnel Features I was impressed with the connector API feature in the system. As stated earlier, over 100 pre-built connectors exist, but you can create a different one if you need to. The connectors are not tied to a specific execution engine but can use Flink, Spark, or the native SeaTunnel one. The plug-in architecture for the connectors reminds me a bit of the . Trino ecosystem Data can be synchronized in batch or real-time, providing various synchronization options. A nifty feature is how it works with JDBC, which supports multi-table or whole database synchronization. This addresses the need for CDC multi-table synchronization scenarios. The runtime process of SeaTunnel is shown in the diagram below: The SeaTunnel runtime flow breaks down as follows: Configure the job information and select the execution engine. The source connector reads the data in parallel and passes it downstream to transform, sink, or directly to the sink. Keep in mind that SeaTunnel is an EL(T) integration platform, as such, it can only do basic data transformations itself: Change the case of data in a column Change a column name Split a column into multiple columns SeaTunnel Jobs A SeaTunnel job, or config file, is described with four possible sections:  env, source, transform, and sink. The transform can be ignored if no transformation is performed. A config file can be written in or format. Borrowing from the SeaTunnel docs, here is a simple example in format: Hocon JSON hocon env {
  job.mode = "BATCH"
}

source {
  FakeSource {
    result_table_name = "fake"
    row.num = 100
    schema = {
      fields {
        name = "string"
        age = "int"
        card = "int"
      }
    }
  }
}

transform {
  Filter {
    source_table_name = "fake"
    result_table_name = "fake1"
    fields = [name, card]
  }
}

sink {
  Clickhouse {
    host = "clickhouse:8123"
    database = "default"
    table = "seatunnel_console"
    fields = ["name", "card"]
    username = "default"
    password = ""
    source_table_name = "fake1"
  }
} While the format is very easy to read and understand, I could see it getting pretty gnarly with large tables. I’ll comment here that, like many open-source projects, the docs are rather lacking, but the project seems to have a pretty active channel based on the time I’ve been in there. Slack SeaTunnel Requirements It’s a Java system, and they say that version 8 or 11 is required but should work with older systems. If you already have Java installed, then you just need to get the plugins you want from their site (or write your own) and set them up in their config file. After that, you create the config file that will manage the job as we described. It’s all pretty straightforward as long as you have the credentials to access your source and destination data repositories. The console will give feedback on what is happening. A is also available for those who want an alternative to the CLI, and that would be my personal preference for using the system, it’s much easier to get visibility, but it also has a lot more steps to install and use it. web interface Summary SeaTunnel is undoubtedly only for some; it comes into play when dealing with lots of data across various data sources and destinations, as I currently see it. I can certainly see situations where it would simplify things, so I’ll keep this project in my bag of tricks. The SeaTunnel folks have this good quick-start guide available , making it simple to give it a try yourself and see how well it can solve your problems. here You can read the other “What the heck” articles at these links: (I was pretty out front on this one.) What The Heck Is DuckDB? (I was out front on this one, too.) What the Heck Is Malloy? (slower, but also growing) What the Heck is PRQL? (growing quickly) What the Heck is GlareDB?

GLAREDB

glaredb.com

1-10 employees

Since 2022

COMPANY RANKING

GLAREDB

EVERGREEN INDEX #4788

Representing this company? Click here to claim and customize this page!

GlareDB WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

GLAREDB

glaredb.com

1-10 employees

Since 2022

COMPANY RANKING

GLAREDB

EVERGREEN INDEX #4788

RELATED COMPANIESRANK

STORIES ABOUT GLAREDB

Representing this company? Click here to claim and customize this page!

GlareDB WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!