The fastest way to ecolabel your products. Helping industrial companies conduct life cycle analyses and get environmental product declarations for their product lines. Interested in joining our team? We're hiring! Apply here: https://www.parqhq.com/careers

Parq

Our team is proud to be part of transforming the way parking spaces work across the whole of Norway and soon the rest of the world, thanks to Parqio and the sharing economy. As the founder of the software development company , I'm always drawn to projects that challenge industries. Parqio, a mobile app revolutionizing parking space sharing, is one such project. Taking inspiration from the likes of and , Parqio's innovative mobile app is changing the way people rent out parking spaces and garages. Voypost Airbnb Uber Today, I invite you behind-the-scenes as I share insights into our collaboration with an environmentally friendly, cheap, and very convenient solution Parqio. Parqio Background When started on their mission to revolutionize parking space management, they faced a few challenges: launch their app fast to validate the idea on the market, implement third-party integrations, test and enhance UX/UI design. It was a daunting task that required a strategic development partner with relevant expertise to help them navigate the mentioned complexities of API, web & mobile app development. Here’s where Parqio started searching for a reliable development team and came across a Berlin-based company, Voypost. Parqio https://www.youtube.com/watch?v=nySwCRD3Lo4&embedable=true Voypost enters the game In November 2020 our team entered the game. There was literally only the idea Parqio C-level had. Ketil Hoigaard (Co-Founder, CEO), Henrik Printz Aamlid (Co-Founder, COO) and Nicholas Kristoffersen, Parqio's ex-CTO shared the vision of how to simplify access to parking spaces, as well as provide an opportunity to rent or lend them. But to make it most convenient for users, automation was needed. For us, the main goal was to turn the vision into reality. We committed to launch an MVP within less than 6 months and deliver high-quality, scalable, easily supported, standardized codebase. With the collaboration initially beginning with just 1 dedicated developer, the Parqio team quickly expanded to include 4 developers from Voypost after a successful MVP launch and a great product demand in the Norwegian market. Currently, we continue working closely  with CTO Benjamin Ochoa and Henrik Printz Aamlidto to drive quality-of-life improvements such as voice interfaces and mobile-friendly guest access tools with 3 developers and 1 project manager engaged with the product. Tech Magic: The Nuts and Bolts of Functionality The product ecosystem consists of 4 main technical aspects. Parqio API is bridging all parts of the Parqio ecosystem. It’s written in and which is shared for all apps & devices listed below. Node.js PostgreSQL Web app called Parqio Admin App to configure the functionality of gates and control access (add/remove residents, grant different access levels to a garage), add gates. This admin panel is built using React.js. Cross-platform Parqio App is used for residents and visitors to open gates, share spaces, find services in the neighborhood, and add automatic gate opening. It uses React Native to be accessible both for iOS/Android users. ParqioBox is a customized IoT device that connects to the garage door controller, somewhat like a router, which connects physical garage door management with Parqio Admin  and the Parqio  App. It is based on Linux/Bash, its firmware uses С/С++. The device logic is written using Node.js and Python integrated into the final firmware version. The journey began with the creation of an MVP, offering a simple design and essential features. Later on, we helped Parqio implement advanced functionality such as: Siri Intent Integration to open doors with voice commands Geofences and mobile push notifications for location-aware prompts to open gates Mapbox integration and live location display Integrating with several 3rd party services such as Twilio and Intercom Custom firmware for Raspberry Pi-based hardware with 4G LTE compatibility Further developing Parqio API for internal and external use, including custom integrations with several Parqio partners iBeacon and Ultra Wide-Band POCs Currently, Parqio cooperates with several partners and allows even more potential suppliers to integrate easily into their system. https://www.youtube.com/watch?v=2THVwIHIoVQ&embedable=true Results in Numbers Henrik Printz Aamlid says: "I’m glad to share my feedback on the partnership with Voypost to make their team shine in this article. The app developed by Voypost has reached around 30.000 users since its launch. The team is very communicative, and they work hard to deliver quality products on time. We are very happy with our collaboration and how they continuously find new ways to add value.” Parqio's performance speaks for itself. Within a year after launch, they acquired more than and garnered over Their partnership with Norway's largest property developer, and the investment from OBOS and among others, totaling over , further validate their success. The company now has its sights set on expanding within the Nordic region and beyond. 60 locations 5,000 users. OBOS , Construct Venture $3.5 million investment Anna, Parqio’s dedicated Project Manager, shared her thoughts on the collaboration: “It's an incredible experience to work on Parqio and witness a huge transformation: from the early days when it was just an idea, to nowadays, with over launched. Our teamwork and agile methodologies of Parqio management, based on product and people-first values, help us achieve all the expected goals and milestones. I'm excited to continue our cooperation and observe the next level of product development because there are a lot of plans for the future.” 200 locations Sum up 4+ people dedicated to the project 5,000+ users in the first year 30,000 total users now by October 2023 200 locations launched $3.5 million in investment raised 3+ years of cooperation is a story of resilience, innovation, and the unwavering belief that great things can be achieved. With Voypost's support, they have achieved remarkable results and are poised for further growth. As Parqio expands its footprint across Europe, it remains a shining example of how technology can revolutionize traditional industries and create new opportunities in the sharing economy. Parqio's journey

How Parqio Hacks Parking Space Management in Norway 

নরওয়েতে পারকিও কীভাবে পার্কিং স্পেস ম্যানেজমেন্ট হ্যাক করে

Wie Parqio das Parkplatzmanagement in Norwegen hackt

Cómo Parqio piratea la gestión de espacios de aparcamiento en Noruega

Comment Parqio pirate la gestion des espaces de stationnement en Norvège

कैसे Parqio नॉर्वे में पार्किंग स्पेस प्रबंधन को हैक करता है

Parqio がノルウェーの駐車スペース管理をハッキングする方法

Parqio가 노르웨이에서 주차 공간 관리를 해킹하는 방법

Como Parqio hackeia o gerenciamento de vagas de estacionamento na Noruega

Как Parqio взламывает управление парковочными местами в Норвегии

Parqio, Norveç'teki Park Alanı Yönetimini Nasıl Hackliyor?

Parqio hack quản lý không gian đỗ xe ở Na Uy như thế nào

Parqio 如何破解挪威的停车位管理

Getting Started With Apache Iceberg and Resources if You Would Like To Go Further

How to Use BigQuery Efficiently: Your Guide to Cost Optimization and Efficient Data Management

A Hands-On Guide to Inverted Indexes: Accelerate Text Searches by 40

In this article, I aim to share my experience standardizing the ETL process within frameworks like Spark and Flink, particularly for those not using data processing frameworks like dbt. Context Data processing is a comprehensive field that encompasses the regular processing of substantial information volumes in both batch and stream modes. These procedures, often referred to as ETL processes, are developed using a variety of programming languages and frameworks. Typically, the main stages of data processing are as follows: Data is collected from various sources for subsequent processing. Reading data: The data undergoes various transformations and manipulations to attain the required structure and format. Transformation: The transformed data is then stored in the target storage. Writing: Schematically, it is represented as shown in the diagram: Challenges After the main stages are implemented, several other stages can be added to enhance and refine the process: This stage involves verifying data compliance with requirements and expectations, including data cleaning to ensure accuracy and completeness. Data quality check: The registration process tracks and documents available data sets. It may also involve data and metadata management for effective data organization. Data registration in the data catalog: This allows visualization of relationships between data assets and ETL processes to understand dependencies. It can help identify potential data quality issues and improve overall data quality. Building data lineage: Additional processing may define the level of data confidentiality and criticality, ensuring proper data protection and security. Data classification: Each of the stages mentioned can be implemented using various technologies and tools, allowing flexibility in choosing the right solution for each specific stage of data processing. Solution Reading data The first step in the ETL process involves reading data from the source. Various types of sources may be used at this stage, such as files, databases, message brokers, and more. The data may also come in different formats, such as CSV, JSON, Parquet, Delta, Hudi, etc. Here is some boilerplate code for reading data from these different sources (implemented as Strategy pattern): from abc import ABC, abstractmethod
from pyspark.sql import SparkSession, DataFrame

# Define the strategy interface
class ReadStrategy(ABC):
    @abstractmethod
    def read_data(self) -> DataFrame:
        pass

# File System Read Strategy - represents a file system reading    
class FileSystemReadStrategy(ReadStrategy):
    def __init__(self, spark: SparkSession, path: str):
        self.spark = spark
        self.path = path
    
    @abstractmethod
    def read_data(self):
        pass
    
# Implement the strategy for CSV
class CSVReadStrategy(FileSystemReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("csv").option("header", "true").load(self.path)


# Implement the strategy for Delta
class DeltaReadStrategy(ReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("delta").load(self.path)

# Implement the strategy for Parquet
class ParquetReadStrategy(ReadStrategy):
    def read_data(self) -> DataFrame:
        return self.spark.read.format("parquet").load(self.path)

# Context class
class DataReader:
    def __init__(self, read_strategy: ReadStrategy):
        self.read_strategy = read_strategy

    def read(self) -> DataFrame:
        return self.read_strategy.read_data() Usage example: spark = SparkSession.builder.appName("our-ETL-job").getOrCreate()
path = "path-to-our-data"
src_df = DataReader(DeltaReadStrategy(spark, path)).read() This approach offers several benefits: It enables the alteration of the reading strategy (source format). It facilitates the addition of new reading strategies without modifying existing code. Each strategy can have its unique parameters. For example, CSV may have a delimiter, while Delta may have a version, etc. Instead of providing a path, we can create a dedicated class, like , which can hold all necessary data reading parameters. ReadConfig It allows the incorporation of additional logic into strategies without affecting the main code. For instance: We can call by default for a file source. input_file_name() If we have a data catalog, we can check whether the table exists there and register it if not. Each strategy can be tested individually. Transformation After reading the data, it must be transformed. This can include various operations like filtering, aggregating, and joining. Since the transformation process can be complex and time-consuming, it's crucial to standardize it for ease of understanding. From my experience, using the method of the DataFrame API is an effective way to standardize this process. This method lets us chain multiple transformations, enhancing code readability. Each transformation can be declared as a separate pure function, performing a specific transformation, and can be tested independently. transform import pyspark.sql.functions as F
from pyspark.sql import DataFrame

desired_columns = ["name", "age", "country"]

def with_country_code(df: DataFrame) -> DataFrame:
    return df.withColumn("country_code", F.when(F.col("country") == "USA", "US").otherwise("Other"))

def with_age_category(df: DataFrame) -> DataFrame:
    return df.withColumn("age_category", F.when(F.col("age") < 18, "child").when((F.col("age") >= 18) & (F.col("age") < 65), "adult").otherwise("senior"))

def filter_adult(df: DataFrame) -> DataFrame:
    return df.filter(F.col("age_category") == "adult")

def select_columns(columns: list) -> callable[[DataFrame], DataFrame]:
    def inner(df: DataFrame) -> DataFrame:
        return df.select(columns)
    return inner

transformed_df = (src_df
                  .transform(with_country_code)
                  .transform(with_age_category)
                  .transform(filter_adult)
                  .transform(select_columns(desired_columns))
                  ) Writing The last stage in the ETL process is writing the transformed data into the target storage, which could be a file system, a database, or a message broker. The loading phase may also incorporate extra steps like data validation, data registration, and data lineage. For this stage, we can apply the same methods as reading data, but with writing strategies. from abc import ABC, abstractmethod
from pyspark.sql import DataFrame, SparkSession

class WriteStrategy(ABC):
    @abstractmethod
    def write_data(self, df: DataFrame):
        pass

class FileSystemWriteStrategy(WriteStrategy):
    def __init__(self, spark: SparkSession, path: str, format: str, mode: str = "overwrite"):
        self.spark = spark
        self.path = path
        self.format = format
        self.mode = mode

    def write_data(self, df: DataFrame):
        df.write.format(self.format).mode(self.mode).save(self.path)

class DataWriter:
    def __init__(self, write_strategy: WriteStrategy):
        self.write_strategy = write_strategy

    def write(self, df: DataFrame):
        self.write_strategy.write_data(df) Usage example: path = "path-to-our-data"
writer = DataWriter(FileSystemWriteStrategy(spark, path, "delta"))
writer.write(transformed_df) Conclusion This implementation standardizes the ETL process, making it more comprehensible and maintainable. It facilitates the addition of new reading and writing strategies without modifying existing code. Each strategy can possess unique parameters, and additional logic can be incorporated into strategies without disrupting the main code. The approach also enables separate testing of each strategy. Moreover, we can introduce additional stages to the ETL process, such as data quality checks, data registration in the data catalog, building data lineage, and data classification.

Dev Standards for Spark-jobs

স্পার্ক-জবসের জন্য ডেভ স্ট্যান্ডার্ড

Entwicklungsstandards für Spark-Jobs

Estándares de desarrollo para trabajos Spark

Normes de développement pour les tâches Spark

स्पार्क-नौकरियों के लिए विकास मानक

Spark ジョブの開発標準

Spark 작업을 위한 개발 표준

Padrões de desenvolvimento para trabalhos Spark

Стандарты разработки для Spark-заданий

Spark işleri için Geliştirme Standartları

Tiêu chuẩn dành cho nhà phát triển cho công việc Spark

Spark 作业的开发标准

Built on Lance, an open-source columnar data format, LanceDB has some interesting features that make it attractive for AI/ML. For example, LanceDB supports explicit and implicit vectorization with the ability to handle various data types. LanceDB is integrated with leading ML frameworks such as and . Cooler still is LanceDB’s fast neighbor search which enables efficient retrieval of similar vectors using approximate nearest neighbor algorithms. All of these combine to create a vector database that is fast, easy to use and so lightweight it can be deployed anywhere. PyTorch TensorFlow LanceDB is capable of querying data in S3-compatible object storage. This combination is optimal for building high-performance, scalable, and cloud-native ML data storage and retrieval systems. MinIO brings performance and unparalleled flexibility across diverse hardware, locations, and cloud environments to the equation, making it the natural choice for such deployments. Upon completion of this tutorial, you will be prepared to use LanceDB and MinIO to joust with any data challenge. What is Lance? The file format is a columnar data format optimized for ML workflows and datasets. It is designed to be easy and fast to version, query, and use for training, and is suitable for various data types, including images, videos, 3D point clouds, audio, and tabular data. Additionally, it supports high-performance random access: with of 100 times faster than Parquet in queries. Lance’s speed is in part the result of being implemented in Rust, and its cloud-native design which includes features like zero-copy versioning and optimized vector operations. Lance Lance reporting benchmarks One of its is the ability to perform vector search, allowing users to find in under 1 millisecond and combine OLAP-queries with vector search. Other production applications for the lance format include edge-deployed low-latency vector databases for ML applications, large-scale storage, retrieval, and processing of multi-modal data in self-driving car companies, and billion-scale+ vector personalized search in e-commerce companies. Part of the appeal of the Lance file format is its compatibility with popular tools and platforms, such as Pandas, , Polars, and . Even if you don’t use LanceDB, you can still leverage the Lance file format in your data stack. key features nearest neighbors DuckDB Pyarrow Built for AI and Machine Learning Vector databases like LanceDB offer distinct advantages for AI and machine learning applications, thanks to their efficient and compute architectures and retrieval of high-dimensional vector representations of data. Here are some key use cases: decoupled storage Natural Language Processing (NLP): Find documents or passages similar to a query based on meaning, not just keywords. This powers chatbot responses, personalized content recommendations, and knowledge retrieval systems. Semantic Search: Understand and answer complex questions by finding relevant text passages based on semantic similarity. Question Answering: Discover latent topics in large text collections, useful for document clustering and trend analysis. Topic Modeling: Computer Vision: Search for similar images or videos based on visual content, crucial for content-based image retrieval, product search, and video analysis. Image and Video Retrieval: Improve the accuracy of object detection and classification models by efficiently retrieving similar training data. Object Detection and Classification: : Recommend similar videos based on the visual content of previously watched videos. Video Recommendation Among the plethora of vector databases on the market, LanceDB is particularly well suited for AI and machine learning, because it supports querying on S3- compatible storage. Your data is everywhere, your database should be everywhere too. Architecting for Success Using MinIO with LanceDB offers several benefits, including: : MinIO’s cloud-native design is built for scale and high-performance storage and retrieval. By leveraging MinIO's scalability and performance, LanceDB can efficiently handle large amounts of data, making it well-suited for modern ML workloads. Scalability and Performance : MinIO is highly available, immutable, and highly durable. This ensures that data stored in MinIO is protected against hardware failures and provides high availability and fault tolerance, which are crucial for data-intensive applications like LanceDB. High Availability and Fault Tolerance : Multi-site, active-active replication enables near-synchronous replication of data between multiple MinIO deployments. This robust process ensures high durability and redundancy, making it ideal for shielding data in mission-critical production environments. Active-active replication The combination of MinIO and LanceDB provides a high-performance scalable cloud-native solution for managing and analyzing large-scale ML datasets. Requirements To follow along with this tutorial, you will need to use . You can install the Docker Engine and Docker Compose binaries separately or together using Docker Desktop. The simplest option is to install Docker Desktop. Docker Compose Ensure that Docker Compose is installed by running the following command: docker compose version You will also need to install Python. You can download Python from . During installation, make sure to check the option to add Python to your system's PATH. here Optionally, you can choose to create a Virtual Environment. It's good practice to create a virtual environment to isolate dependencies. To do so, open a terminal and run: python -m venv venv To Activate the virtual environment: On Windows: .\venv\Scripts\activate On macOS/Linux: source venv/bin/activate Getting Started Begin by cloning the project from . Once done, navigate to the folder where you downloaded the files in a terminal window and run: here docker-compose up minio This will start up the MinIO container. You can navigate to ‘ ’ to take a look at the MinIO console. http://172.20.0.2:9001 Log in with the username and password . minioadmin:minioadmin Next, run the following command to create a MinIO bucket called . lance docker compose up mc This command performs a series of (mc) commands within a shell. MinIO Client Here's a breakdown of each command: ;: This command repeatedly attempts to configure a MinIO host named with the specified parameters (endpoint, access key, and secret key) until successful. During each attempt, it echoes a waiting message and pauses for 1 second. until (/usr/bin/mc config host add minio http://minio:9000 minioadmin minioadmin) do echo '...waiting...' && sleep 1; done minio This command forcefully removes (deletes) all contents within the bucket in MinIO. /usr/bin/mc rm -r --force minio/lance;: lance : This command creates a new bucket named in MinIO. /usr/bin/mc mb minio/lance; lance : This command sets the policy of the bucket to public, allowing public read access. /usr/bin/mc policy set public minio/lance; lance : This command ensures that the script exits with a status code of 0, indicating successful execution. exit 0; LanceDB Unfortunately, LanceDB does not have native S3 support, and as a result, you will have to use something like boto3 to connect to the MinIO container you made. As LanceDB matures we look forward to native S3 support that will make the user experience all the better. The sample script below will get you started. Install the required packages using pip. Create a file named requirements.txt with the following content: lancedb~=0.4.1
boto3~=1.34.9
botocore~=1.34.9 Then run the following command to install the packages: pip install -r requirements.txt You will need to change your credentials if your method of creating the MinIO container differs from the one outlined above. Save the below script to a file, e.g., . lancedb_script.py import lancedb
import os
import boto3
import botocore
import random

def generate_random_data(num_records):
    data = []
    for _ in range(num_records):
        record = {
            "vector": [random.uniform(0, 10), random.uniform(0, 10)],
            "item": f"item_{random.randint(1, 100)}",
            "price": round(random.uniform(5, 100), 2)
        }
        data.append(record)
    return data

def main():
    # Set credentials and region as environment variables
    os.environ["AWS_ACCESS_KEY_ID"] = "minioadmin"
    os.environ["AWS_SECRET_ACCESS_KEY"] = "minioadmin"
    os.environ["AWS_ENDPOINT"] = "http://localhost:9000"
    os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

    minio_bucket_name = "lance"

    # Create a boto3 session with path-style access
    session = boto3.Session()
    s3_client = session.client("s3", config=botocore.config.Config(s3={'addressing_style': 'path'}))

    # Connect to LanceDB using path-style URI and s3_client
    db_uri = f"s3://{minio_bucket_name}/"
    db = lancedb.connect(db_uri)

    # Create a table with more interesting data
    table = db.create_table("mytable", data=generate_random_data(100))

    # Open the table and perform a search
    result = table.search([5, 5]).limit(5).to_pandas()
    print(result)

if __name__ == "__main__":
    main() This script will create a Lance table from randomly generated data and add it to your MinIO bucket. Again, if you don’t use the method in the previous section to create a bucket you will need to do so before running the script. Remember to change the sample script above to match what you name your MinIO bucket. Finally, the script opens the table, without moving it out of MinIO, and uses Pandas to do a search and print the results. The result of the script should look similar to the one below. Remember that the data itself is randomly generated each time. vector      item  price  _distance
0  [5.1022754, 5.1069164]   item_95  50.94   0.021891
1   [4.209107, 5.2760105]  item_100  69.34   0.701694
2     [5.23562, 4.102992]   item_96  99.86   0.860140
3   [5.7922664, 5.867489]   item_47  56.25   1.380223
4    [4.458882, 3.934825]   item_93   9.90   1.427407 Expand on your Own There are many ways to build on this foundation offered in this tutorial to create performant, scalable and future-proofed ML/AI architectures. You have two cutting-edge and open-source building blocks in your arsenal – MinIO object storage and the LanceDB vector database –  consider this your winning ticket to the ML/AI . tournament Don’t stop here. LanceDB offers a wide range of and tutorials to expand on what you’ve built in this tutorial including a recently announced Udacity course on . Of particular interest is recipe to chat with your documents. We are all for breaking down barriers to getting the most from your data. recipes Building Generative AI Solutions with Vector Databases this Please show us what you’re building and should you need guidance on your noble quest don’t hesitate to email us at hello@minio.io or join our round table on . Slack

LanceDB: Your Trusted Steed in the Duel Against Data Complexity

ল্যান্সডিবি: ডেটা জটিলতার বিরুদ্ধে দ্বন্দ্বে আপনার বিশ্বস্ত স্টাইড

LanceDB: Ihr vertrauenswürdiges Pferd im Kampf gegen die Datenkomplexität

LanceDB: su corcel de confianza en el duelo contra la complejidad de los datos

LanceDB : votre monture de confiance dans le duel contre la complexité des données

लांसडीबी: डेटा जटिलता के विरुद्ध द्वंद्व में आपका विश्वसनीय घोड़ा

LanceDB: データの複雑さとの戦いにおいて信頼できる馬

LanceDB: 데이터 복잡성과의 결투에서 신뢰할 수 있는 군마

LanceDB: seu corcel confiável no duelo contra a complexidade dos dados

LanceDB: ваш надежный помощник в борьбе со сложностью данных

LanceDB: Veri Karmaşıklığına Karşı Düelloda Güvenilir Atınız

LanceDB: Chiến mã đáng tin cậy của bạn trong cuộc đấu tranh chống lại sự phức tạp của dữ liệu

LanceDB：对抗数据复杂性的值得信赖的骏马

What data management in 2024 and beyond will look like hangs on one question. Can open data formats lead to a best-of-breed data management platform? It will take Interoperability across clouds and formats, as well as on the semantics and governance layer. Sixth Platform. Atlas. Debezium. DCAT. Egeria. Nessie. Mesh. Paimon. Transmogrification. This veritable word soup sounds like something that jumped out of a role-playing game. In reality, these are all terms related to data management. These terms may tell us something about the mindscape of people involved in data management tools and nomenclature, but that’s a different story. Data management is a long-standing and ever-evolving practice. The days of “Big Data” have long faded and data has now taken a back seat in terms of hype and attention. is the new toy people are excited about, shining a light on AI and machine learning for the masses. Generative AI Data management may seem dull in comparison. However, there is something people who have been into AI before it was cool understand well, and people who are new to this eventually realize too: AI is only as good as the data it operates on. There is an evolutionary chain leading from . Some organizations were well aware of this a decade ago. Others will have to learn the hard way today. data to analytics to AI Exploring and understanding how, where, and when data is stored, managed, integrated and used as well as related aspects of data formats, interoperability, semantics, and governance may be hard and unglamorous work. But it is what generates value for organizations, along with -like word soups. RPG Peter Corless and Alex Merced understand this. Their engagement with data goes back to way before their current roles as Director of Product Marketing at and Developer Advocate at , respectively. We caught up to talk about the state of data management in 2024, and where it may be headed. StarTree Dremio https://www.youtube.com/watch?v=Meq247f31-A&embedable=true The sixth data platform For many organizations today, data management comes down to handing over their data to one of the “Big 5” data vendors: , Azure, and , plus Snowflake and Databricks. However, analysts David Vellante and George Gilbert believe that the need of modern data applications coupled with the evolution of open storage management may lead to the emergence of a “sixth data platform”. Amazon Microsoft Google The notion of the “sixth data platform” was the starting point for this conversation with Corless and Merced. is that open data formats may enable interoperability, leading the transition away from vertically integrated vendor-controlled platforms towards independent management of data storage and permissions. The sixth data platform hypothesis It’s an interesting scenario and one that would benefit users by forcing vendors to compete for every workload based on the business value delivered, irrespective of lock-in. But how close are we to realizing this? To answer this question, we need to examine open data formats and their properties. In turn, to understand data formats a brief historical overview is needed. https://www.youtube.com/watch?v=j7s8xSTxkYk&embedable=true We might call this table stakes, and it’s not just a pun. Historically, organizations would resort to using data warehouses for their analytics needs. These were databases specifically designed for analytics, as opposed to transactional applications. Traditional data warehouses worked, but scaling them was an issue as Merced pointed out. Scaling traditional data warehouses was expensive and cumbersome because it meant buying new hardware with storage and computing bundled. This was the problem Hadoop was meant to solve when introduced in the early 2010s, by separating storage and compute. Hadoop made scaling easier, but it was cumbersome to use. This is why an SQL interface was eventually created for Hadoop via , making it more accessible. Hive used metadata and introduced a protocol for mapping files, which was what Hadoop operated on, to tables, which is what SQL operates on. That was the beginning of open data formats. Apache Hive Data lakes, data lakehouses, and open data formats Hadoop also signified the emergence of the data lake. Since storage was now cheap and easy to add, that opened up the possibility of storing all data in one big Hadoop store: the data lake. Eventually, however, , which is the realm the “Big 5” operate on. Hadoop and on-premise compute and storage gave way to the cloud “The idea of a data lakehouse was, hey, you know what? We love this decoupling of compute and storage. We love the cloud. But wouldn’t it be nice if we didn’t have to duplicate the data and we could just operate over the data store you already have, your data lake, and then just take all that data warehouse functionality and start trying to move it on there?”, as Merced put it. However, as he added, that turned out to be a tall order, because there were elements of traditional data management missing. Other cloud-based data management platforms introduced notions like automated sharding, partitioning, distribution, and replication. That came as part of the move away from data centers and file systems towards the cloud and APIs to access data. But for data lakehouses, these things are not a given. This is where open data formats such as and come in. They all have some things in common, such as the use of metadata to abstract and optimize file operations and being agnostic about storage. However, Hudi and Delta Lake both continued to build on what Hive started, while Iceberg departed from that. Apache Iceberg, Apache Hudi, Delta Lake https://www.youtube.com/watch?v=043GGYTZCbk&embedable=true A detailed open data format comparison is a nuanced exercise, but perhaps the most important question is: Does interoperability exist between open data formats, and if yes, on what level? For the sixth data platform vision to become a reality, Interoperability should exist among clouds and formats, as well as on the semantics and governance level. Interoperability of the same format between different cloud storage vendors is possible, but not easy. It can be done, but it’s not an inherent feature of the data formats themselves. It’s done on the application level, either via custom integration or by using something like Dremio. And there will be egress and network costs from transferring files outside the cloud where they reside. Interoperability of different formats is also possible, but not perfect. It can be done by using a third-party application, but there are also a couple of other options like or . Neither works 100%, as Delta Uniform batches transactions and OneTable is for migration purposes. Delta Uniform OneTable As Merced noted, creating a solution that works 100% would probably incur a lot of overhead and complexity as metadata would have to be synchronized across formats. Merced thinks these solutions are motivated at least partially by Iceberg’s growth and the desire to ensure third-party tools can work with the Iceberg ecosystem. To federate or not to federate? What’s certain is that there’s always gonna be sprinkles of data in other systems or multiple clouds. That’s why federation and virtualization that can work at scale is needed, as Corless noted. Schema management is complicated enough on one system, and trying to manage changes across three different systems would not make things easier. Whether data is left where it originally resides and used via federated queries, or ingested in one storage location is a key architectural question. As such, there are tradeoffs involved that need to be understood. Even within federation, there are different ways to go about it – Dremio, GraphQL, Pinot, Trino, and more. “How do data consumers want to size up these problems? How can we give them predictive ways to plan for these trade-offs? They could use federated queries, or a hybrid table of real time and batch data. We don’t even have a grammar to describe those kinds of hybrid or complex data products these days”, Corless said. Corless has a slightly different point of view, focusing on real-time data and processing. This is what StarTree does, as it builds on t. Pinot is a real-time distributed OLAP datastore, designed to answer OLAP queries with low latency. Apache Pino Corless also noted that the choice of data format depends on several parameters. Some formats are optimized for in-memory use, like Others like are optimized for disk storage. Apache Arrow. Apache Parquet There are attempts to have common representations of data both in-memory and on storage, motivated by the need to leverage tiered storage. Tiered storage may utilize different media, ranging from in-memory to SSD to some sort of blob storage. “People want flexibility in where and how they store their data. If they have to do transmogrification in real time, that largely defeats the purpose of what they’re trying to do with tiered storage. Is there something like the best, most universal format? Whenever you’re optimizing for something, you’re optimizing away from something else. But I’m very eager to see where that kind of universal representation of data is taking us right now.”, Corless said. Semantics and governance Regardless of where or how data is stored, however, true interoperability should also include semantics and governance. This is where the current state of affairs leaves a lot to be desired. As Merced shared, the way all table formats work is that the metadata has information about the table, but not about the overall catalog of tables. That means they are not able to document the semantics of tables and how they relate to each other. A standard format of doing that doesn’t exist yet. Merced also noted that there is something that addresses this gap, but it only works for Iceberg tables. is an open-source platform that creates a catalog, thus making tracking and versioning Iceberg tables and views possible. Project Nessie Nessie also incorporates elements of governance, and Merced noted that Nessie will eventually have Delta Lake support too. Databricks on its part offers which works with Delta Lake, but it’s a proprietary product. There is no lack of data catalog products in the market, but none of that can really be considered the solution to semantics and data interoperability. Unity Catalog Corless on his part noted that there a standard called DCAT. DCAT, which stands for Data Catalog Vocabulary, is an RDF vocabulary designed to facilitate interoperability between data catalogs published on the Web. DCAT was recently updated to v.3, it’s been around for over a decade, and it’s precisely aimed at interoperability. https://www.youtube.com/watch?v=Pb53L77_nS8&embedable=true The fact that DCAT is not used widely has probably more to do with vendors reinventing the wheel and/or aiming for lock-in, as well as clients not being more proactive in requiring interoperability standards than with DCAT itself, as per Corless. Unfortunately, it seems like what data is to AI, governance is to data for most organizations: a distant second at best. Forty-two percent of data and analytics leaders do not assess, measure or monitor their data and analytics governance, according to a . Those who said they measured their governance activity mainly focused on achieving compliance-oriented goals. 2020 Gartner survey The onset of . was an effort to standardize governance for data lakes leveraging DCAT and other metadata vocabularies. Today, however, both Data Lakes and Atlas seem to have fallen out of fashion. GDPR in 2018 marked the opportunity for governance to leverage metadata and semantics to rise up Apache Atlas A new project called was spun out of Atlas, aiming to address more than data lakes. Furthermore, there is another open standard for metadata and data lineage called and several lineage platforms that support it, including Egeria. Egeria OpenLineage Conclusion So, what is the verdict? Is the sixth data platform possible for data management in 2024 or is it a pipedream? Merced thinks that we’re already starting to see it. In his view, using open formats like Apache Iceberg, Apache Arrow, and Apache Parquet can create a greater level of interoperability. This interoperability is still imperfect, he noted, but it’s a much better state of things than in the past. Dremio’s thesis is to be an open data lakehouse, not just operating on the data lake, but across tools and data sources. Corless thinks that we’re going to see a drive toward clusters of clusters, or systems of systems. There’s going to be a reinforced drive towards the automation of integration. More like Lego bricks that snap together easily, rather than Ikea furniture that comes with a hex wrench. But for that to happen, he noted, we’ll need a language and a grammar so that systems and people can both understand each other. The ingredients for a sixth data platform are either there, or close enough. As complexity and fragmentation explode, a language and grammar for interoperability sounds like a good place to start. Admittedly, interoperability and semantics are hard. Even that, however, may be more of a people and market issue than a technical one. DCAT and OpenLineage are just some of the vocabularies out there. Even things as infamously hard to define as data mesh and data products have a vocabulary of their own – the . Data Product Descriptor Specification Perhaps then the right stance here would be cautious optimism. Also published . here

Data Management in 2024: Will Open Data Formats Shape a “Sixth Platform”?

Looks like it’s part 2 of my series on Data Quality! In a , I explored Airbnb’s strategy for enhancing data quality through incentives. They implemented a single score and clear scoring criteria to establish a common understanding between data producers and consumers, fostering a genuine sense of ownership. previous post https://hackernoon.com/data-quality-score-one-score-to-rule-them-all?embedable=true Now, Lyft is taking a distinct approach, not attempting the same thing differently, but rather focusing on different aspects of data quality. And, Lyft’s strategy complements Airbnb’s efforts. While I regard Airbnb’s DQ score (or any similar score) as an effective means to consolidate various attempts at elevating data quality, Lyft is tackling this challenge from a different angle. The Airbnb’s DQ score serves as a valuable tool for providing a concrete visualization of data quality. In essence, any initiative to enhance data quality should manifest a discernible impact on this score. On the other hand, Lyft presents one possible initiative to proactively enhance quality by testing and validating data against specific quality criteria. https://eng.lyft.com/from-big-data-to-better-data-ensuring-data-quality-with-verity-a996b49343f6?source=post_page-----415baad6c95b--------------------------------&embedable=true Fundamentally, it’s a different point in the data quality lifecycle. Introducing a mechanism to improve quality necessitates the ability to measure it initially. So, while ’s focus lies on and data quality, relying on the producer’s interest to enhance this quality and “look good,” Lyft takes a different approach. Lyft places emphasis on actively and data quality, providing both producers and consumers with the means to effectively improve and control the quality. Airbnb measuring observing testing validating Collectively, these approaches provide a comprehensive strategy to address and enhance throughout its . data quality lifecycle For this reason, I was particularly interested in taking a closer look at Lyft’s approach. Another factor that intrigued me is testing, more specifically, contract testing, which has been used for many years now in basic software engineering with the emergence of microservice architecture. Data contracts, however, are something more recent in the domain of data engineering and are seen as the pinnacle or one of the ultimate steps to put in place on the path to building high-quality data pipelines. This is why I wanted to examine Lyft’s approach in more detail and explore some potential parallels. As mentioned, the approaches taken by Airbnb and Lyft are complementary and aim to achieve the same goal: improving data quality. has developed the DQ score, which focuses on measuring and enhancing 4 distinct : Airbnb aspects of data quality DQ Score has guiding principles, including full coverage, automation, actionability, multi-dimensionality, and evolvability. It has dimensions like Accuracy, Reliability, Stewardship, and Usability. Lyft’s Verity is a platform designed to enhance data quality across 5 dimensions Defines data quality as the measure of how well data can be used as intended, covering aspects like . semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness It’s easy to draw parallels between the 5 data quality aspects improved by Lyft’s Verity and the 4 data quality dimensions measured by the Airbnb’s DQ score. For example, aspects like Timeliness would certainly contribute to the DQ score’s Reliability, while Accuracy would be dependent on semantic correctness, completeness, and uniqueness of the data. The Usability score, on the other hand, is influenced by the consistency of the data among other factors. Lyft’s Verity focuses on defining related to . It follows a test-first and validation approach, whereas Airbnb’s unified DQ score emphasizes evaluating data quality through various dimensions. checks semantic correctness, consistency, completeness, uniqueness, well-formedness, and timeliness If we wanted to incorporate the DQ score into this last schema, it would be on the side of the Alert/Debug steps. Airbnb’s DQ score uses different signals to assess data quality across the . Accuracy, Reliability, Stewardship, and Usability aspects We also had a set of input signals that more directly measure quality (Midas certification, data validation, bugs, SLAs, automated DQ checks, etc.), whereas others were more like proxies for quality (e.g., valid ownership, good governance hygiene, the use of paved path tooling). As discussed earlier, there are likely overlaps between Airbnb’s DQ score and Verity. While Airbnb focuses on pushing data quality to the right, emphasizing measurement and scoring, Lyft’s Verity takes a proactive approach by shifting the check definition configurations, testing, and validation processes to the left, emphasizing proactive improvement of data quality. Now, my primary interest lies to the left, in the check definition configurations, testing, and validation aspects. How Lyft integrates data quality testing into its processes? Let’s first examine the execution paths. Currently, Lyft’s Verity is primarily focused on ensuring the quality of data stored in its Hive data warehouse. However, there are plans to expand its capabilities to support other data sources in the future. Note that while they refer to Hive as a data workhouse, they actually utilize it as a hybrid data storage solution, operating as a data warehouse for structured, processed, and cleansed data (silver layer) while also serving as a data lake for raw event data (bronze layer). Poor data quality in Hive caused tainted experimentation metrics, inaccurate machine learning features, and flawed executive dashboards. Verity’s checks can be integrated into an Airflow DAG to ensure that only high-quality raw data is processed and stored in Hive as derived data. Data producers and consumers can define their data quality checks and verify the data when it is produced or before it is consumed inside or . Airflow Flyte … The VerityAirflowOperator can be used in a blocking fashion to halt a DAG upon a check failure, preventing bad data from ever reaching production. This utilizes the “ ” pattern: we create data in a staged schema, verify the data with a blocking operator, then promote it to production if it passes quality checks. Stage-Check-Exchange Checks can also be performed manually or scheduled automatically to verify both raw and derived data. Verity Scheduled Checks are isolated from any data orchestration engine, so they still run even if Airflow or Flyte are completely down. This remedies a common problem of checks not alerting because the Airflow Task never ran. So, there are essentially 3 primary ways to trigger a check: as part of an Airflow DAG, manually, or scheduled through the Verity platform/UI. I don’t believe that current checks can be integrated into real-time streaming pipelines (such as Flink + Kafka) to validate for example data as it enters Hive (or even earlier). Implementing this type of real-time check would enable prompt detection of discrepancies, leading to reduced storage and processing costs and overall data quality enhancement. Well, to be thorough, Verity checks are managed through an API server, which could be used to trigger checks programmatically through some APIs. Verity API Server — This service handles all the external APIs regarding running checks as well as persisting and retrieving their results. The API Server does not execute any checks, but rather writes a message to our Check Queue, which utilizes (SQS). SimpleQueueService So, maybe, you could potentially trigger these jobs in a more real-time fashion, such as from a streaming job or even, as a long stretch, by integrating with Hive change data capture (CDC). However, when executed outside of Airflow, these jobs would not be able to block the data processing job; instead, they would generate asynchronous alerts pushed to the Check Queue. Some consumers would prefer to have data processing delayed when a check fails, while others would rather proceed and receive an alert. Now, let’s look at these data quality tests. Here’s an example that checks whether is never null. This is accomplished through a combination of query and condition components. rider_events.session_id core rider events session_id is not null: # check name
  metadata:
    id: 90bde4fa-148b-4f06-bd5f-f15b3d2ad759
    ownership_slack: #dispatch-service-dev
    tags: [rides, core-data, high-priority]
  query:
    type: dsl
    data_source_id: hive.core.rider_events
    filters:
      - session_id = null
  condition:
    type: fixed_threshold
    max: 0
  notifier_group:
    pagerduty_policy: dispatch-service
    email: dispatch-service@lyft.pagerduty.com Verity is primarily focused on defining and enforcing data quality checks rather than defining complete data schemas. Schema validation is not a novel concept. Several methods exist for defining event data schemas in event-based systems, such as JSON Schema, Protocol Buffers, Avro, or storage formats like Parquet. The optimal choice depends on your tech stack, usage, and specific requirements. While data schemas are valuable for defining the overall structure of data objects or table rows, they fall short in capturing more sophisticated validation checks specific to consumers, such as data distribution, business rules, SLAs, and thresholds. Data contracts go beyond schema validation, which focuses on identifying . I personally find that JSON Schema offers here a more suitable and readable option, effectively separating these structural and syntactical validation capabilities from serialization or storage concerns. syntactical errors However, to address and enhance data accuracy, having an effective means of defining data checks allows for defining the other facet of data contracts. semantic errors This is where Verity DSL comes into play. Beyond semantical validation, data contracts offer another crucial aspect that deserves attention. From a syntactical standpoint, validation checks remain consistent regardless of the consumer or producer involved. The set of validation rules is not tied to any specific consumer or producer and can be defined once and for all as a single schema. However, the Verity data contract DSL offers a finer granularity defining small independent rules, which is particularly well-suited for this context: the semantic meaning and usage of the data vary depending on the specific consumer. Additionally, not all consumers need to utilize all of the properties of an object. Their expectations differ. This doesn’t imply they are contradictory (which would certainly be an issue), but rather complementary and distinct ones. Therefore, allowing all consumers to establish unique rules that, when combined collaboratively, can provide a comprehensive understanding of the semantic significance of data quality is quite crucial. And it’s this aspect that particularly resonates with me. Bear with me, this may seem like a stretch, but from my perspective, it’s worth mentioning. :) collaborative Data exchange enables different teams (producers and consumers) to collaborate effectively. Establishing a shared understanding of these data exchanges is paramount, just like APIs in traditional software development. In microservice architectures, a known as consumer-driven contracts (CDC) has emerged, where consumers define the expected behavior of APIs provided by producers. Producers are responsible for verifying these contracts before releasing new versions. collaborative testing approach I think data contracts share a similar collaborative spirit. Even though data validation is performed on real data, rather than at release time, and does not block releases, it’s based on cooperation and encourages teamwork between data producers and consumers. I strongly believe that this collaborative approach is key to improving data quality and should be further integrated into the process. Well, I’m a big fan of drawing parallels… Notice actually that this collaborative aspect is something mentioned also as part of Lyft’s verity charter. The VerityUI provides a streamlined data discovery experience via the Verity Homepage. Our full-text search on the Check Definition Metadata lets users see all the checks currently being enforced and their Check Results. This has useful aggregations like owning team, table name, and tags. I’m not entirely clear on how data contract issues are shared between consumers and producers through the Verity platform UI, but I definitely recognize the significance of collaboration through the dashboards: The producer of a data product interface can confidently ensure that they’re not inadvertently causing downstream breakages that they hadn’t anticipated. The consumer of the interface can rest assured that their reliance on the interface is not and will not be compromised. While Verity is a remarkable tool for defining data quality checks, it’s unfortunately not open-source. Fortunately, there’s another open-source data quality framework called Soda Core that provides similar functionality. Soda Core is a free and open-source command-line tool and Python library that enables data engineers to test data quality. It utilizes user-defined input to generate SQL queries that run checks on datasets in a data source to find invalid, missing or unexpected data. When checks fail, they surface the data that you defined as “bad” in the check. https://medium.com/@seckindinc/soda-core-the-simplest-open-source-data-reliability-tool-f35ae5b576d9?source=post_page-----415baad6c95b--------------------------------&embedable=true During a scan of a dataset, Soda Core evaluates the predefined checks to identify invalid, missing, or unexpected data. Here’s the equivalent check using Soda.core syntax for the Verity DSL test that was previously defined. name: rider_events_session_id_check
source: hive
query: SELECT * FROM rider_events WHERE session_id IS NULL;
raise_alert: true
threshold: 10
action: slack
message: "There are more than 10 rows that are null for the 'session_id' property in the 'rider_events' table. Please investigate this issue." soda run --check checks/rider_events_session_id_check.yaml Soda Core is a powerful tool for ensuring the quality of your data. It can help you to identify and fix data problems early, before they can cause issues for your business. It’s worth noting that Soda Core can also facilitate data quality checks for streaming data by seamlessly integrating with Spark DataFrames. While Verity’s data quality checks for Hive are applied to static datasets, the checks for streaming data need to be more lightweight and efficient. Data would typically be processed in small batches of events, with very low latency, making them suitable for real-time checks and specific use cases like anomaly detection. Finally, let’s mention that there are other data validation tools available, such as DeeQu, Great Expectations, … As we wrap up, I hope you have a clearer understanding of the steps you can take to enhance your data quality journey. We’ve seen two distinct approaches to data quality improvement, each with its own strengths and methodologies. One focuses on increasing visibility and observability, motivating data producers to raise the quality bar. The other prioritizes elevating the quality standard through a testing and validation-first approach. Both are complementary. Verity is not merely a domain-specific language (DSL) for defining data checks; it’s a centralized platform that empowers data practitioners to collaborate effectively. This platform helps producers and consumers align on data quality expectations, including format, structure, and accuracy. Verity’s data contract management capabilities could (is?) further enhanced by integrating with a broader set of features, such as metadata management and discovery, to address more sophisticated data quality needs. Similar to Airbnb’s DQ score, Lyft’s Verity fosters a collaborative feedback loop between data producers and consumers. By incentivizing and empowering each team to take ownership of data quality, Verity cultivates a supportive environment where data quality continuously improves. , Found this article useful? Follow me on Linkedin , and ! Please 👏 this article to share it! Hackernoon Medium Also published . here

Advancing Data Quality: Exploring Data Contracts with Lyft

ডেটা গুণমান উন্নত করা: Lyft-এর সাথে ডেটা চুক্তিগুলি অন্বেষণ করা

Verbesserung der Datenqualität: Erkundung von Datenverträgen mit Lyft

Mejora de la calidad de los datos: exploración de los contratos de datos con Lyft

Améliorer la qualité des données : explorer les contrats de données avec Lyft

डेटा गुणवत्ता को आगे बढ़ाना: Lyft के साथ डेटा अनुबंध की खोज करना

データ品質の向上: Lyft とのデータ契約の検討

데이터 품질 향상: Lyft를 통한 데이터 계약 탐색

Avançando na qualidade dos dados: explorando contratos de dados com Lyft

Повышение качества данных: изучение контрактов на передачу данных с Lyft

Veri Kalitesini Geliştirme: Lyft ile Veri Sözleşmelerini Keşfetmek

Nâng cao chất lượng dữ liệu: Khám phá hợp đồng dữ liệu với Lyft

提高数据质量：与 Lyft 探索数据合同

PARQ

parqhq.com

11-50 employees

Since 2022

COMPANY RANKING

PARQ

EVERGREEN INDEX #7444

Representing this company? Click here to claim and customize this page!

Parq WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

PARQ

parqhq.com

11-50 employees

Since 2022

COMPANY RANKING

PARQ

EVERGREEN INDEX #7444

RELATED COMPANIESRANK

STORIES ABOUT PARQ

Representing this company? Click here to claim and customize this page!

Parq WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!