Data Lake develops and implements cutting-edge technologies to help advance ethical medical research, and improve access to treatment.

Data Lake

datalake

Welcome to HackerNoon’s interview series. This section is curated by our Editorial team to spotlight bold and disruptive startups across the globe. If you’ve been nominated, and answer these questions Startups-Of-The-Year create a brand profile here. Hey Hackers, EU-funded startup Data Lake has been nominated in HackerNoon's annual Startup of the Year awards in Warsaw, Poland! We would be beyond grateful if you could vote for us here: https://hackernoon.com/startups/europe/europe-warsaw-poland Read more about us below to understand the amazing things we’re working on, and why we believe we deserve your vote. Meet Data Lake Founded by two medical doctors, is an EU-funded startup working to solve two of the biggest issues in medical research today: access to - and bias in - medical data. Data Lake Humanity is on the brink of some incredible developments in medical technologies and treatments - from AI algorithms detecting cancer to nano-robotics and smart sensors, we are looking at a future in which patient outcomes are hugely impacted by medical technology. Having access to high-quality and representative data, however, is of vital importance to the design, training, and implementation of these technologies. Up until recently, however, there was no way for researchers to legally and ethically access the datasets needed to build these incredible developments. Thanks to new EU legislation and our tools, we have created a way for researchers to do just that! Based on ethically-gathered digital consents, Data Lake enables patients to provide access to their medical data to the research entities and startups who are building our medical future. Along with the and more than 40 patient and healthcare organizations, we have that will soon allow anyone in the world to donate their medical data to science. Donate Your Data Foundation launched a global medical data donation platform My Role My name is Wojciech Sierocki, co-founder and CEO of Data Lake. I am a medical doctor by training but have previously founded several startups including ‘Treatment with a Mission’, at the time one of Poland’s largest NGOs operating in Africa. From the start of the COVID pandemic, I worked as a physician in the Emergency Department of the HCP Hospital in Poznan, Poland. While being able to make an immediate impact in someone’s life as a doctor was something I found immensely satisfying, seeing the impact of data on COVID outcomes I soon realized that my greater potential resided in building systems that could impact hundreds of thousands - if not millions - of people. Thus, along with my co-founder Dr. Ligia Kornowska, the idea for an international medical data donation program and Data Lake was born. How We're Disrupting the Medical Research Industry We have built an ethical medical data access layer that allows medical researchers (who are performing pro-patient research) to study large amounts of medical data that has previously never been studied. Based on the informed consent of patients who have the right to revoke it at any time, we can connect researchers to multi-source data sets that can be incrementally and regularly updated. This can take various forms, including the creation of privacy-focused patient registries that allow researchers to connect with patients suffering from ultra-rare diseases, as well as enabling a new era of digital clinical trials. Standing Out from The Crowd Securing access to medical data has been attempted - and failed - before. This has primarily been due to two factors - first, previous attempts have required patients and the general public to gain access to and then upload their medical data. For anyone who has ever tried to get their medical files, they’ll know just how difficult a proposition that is. Additionally, previous attempts have not sufficiently taken into account the aspects of privacy and transparency, often asking people to trust them regarding how and when their data is being used. We have solved both of these issues, using technology that allows for direct integration with hospitals for the transfer of medical records based on the express consent of patients, given digitally and recorded with distributed ledger technology. In short, we do all the heavy lifting by taking on both all of the legal and operational burdens. What really makes our approach different, however, is the focus on openness and transparency. We don’t want patients to have to trust that their consent (or revocation of consent) has been received, and therefore we chose blockchain technology to ensure a verifiable and trust-less system. By recording operations of consent on a public blockchain - without personally-identifiable information of course - we have removed the need for trust from the system entirely, allowing patients to know beyond a shadow of a doubt what their consent status is, and whether their medical data is being used for research or not. Our system not only respects patient agency and privacy, but it reduces the risk for researchers who know absolutely whether they have permission to study a dataset or not. Our Predictions/Thoughts on the HealthTech Industry in 2023 As with many industries, the healthcare industry will become increasingly reliant upon and driven by data in the coming years. A.I. algorithms, machine learning, and smart sensors are already online and assisting in the diagnosis and treatment of diseases, and with continuous and parallel innovations in nanorobotics and 3D printing, we are looking at a Science Fiction level of healthcare being possible within the next few years. Our main prediction - and something we are working to ensure - is that the ethical sourcing, storage, and analysis of data will become an increasing trend. Before we started and before key EU legislation was put in place, the idea of “consent-based data donation” was foreign to many people in the healthcare industry. Thanks in part to our efforts, we are hearing it talked about more and more at conferences and in the media. We believe that engaging people in what happens with their medical data and ensuring they are stakeholders is key to unlocking a new wave of medical innovation. We are convinced that you can have both privacy and innovation, and we believe the coming year will prove us right. What word defines the state of HealthTech in 2023? It’s hard to define in just one word, but we’ll give you three: “exciting, but critical.” The technological innovations we’ve talked about previously are incredibly exciting when you consider their potential for improving quality of life, patient outcomes, and ultimately, saving lives. We’re not exaggerating when we say we’re on the brink of being able to cure and treat many diseases previously considered unbeatable. But how we build these technologies and algorithms - particularly when it comes to the data we are using to create and train them - is of crucial importance. It’s why we’re working not just on the technological side of things, but also engaging on the ethics and rules for practical implementation of these exciting developments, such as my co-founder being a leading contributor to the . Whitepaper on AI in Clinical Practice Why we decided to participate in HackerNoon's Startup of the Year awards To be honest, one of our community members must have signed us up because it was a surprise to us! But a very pleasant one, because we believe that our technology and the idea of medical data donation have the power to save lives, and we want as many people out there to be aware of this growing movement. Final Thoughts We’re very thankful for the nomination and would like to thank everyone who has voted for us, and we hope that if you haven’t already you will consider doing so! We would also love for you to check out as well and sign up for the waiting list to become a medical data donor! Best of luck to everyone! https://donateyourdata.io Click here: Vote for us today!

Startups of the Year 2023: Data Lake - The Startup That Is Improving Access to Medical Research

Many of us have made changes to data based on requirements that later evolve. By the time we realize it, it becomes impossible to roll back. Change isn't just a constant; it's an essential part of managing data that demands a sophisticated approach. Enter the Git-like functionality of Dremio's data catalog Nessie. Just as Git has become fundamental to software development, data engineers need similar tools to work in parallel, compare data versions, promote changes to production and roll back data when needed. Nessie provides data engineers with a Git-like version control system for managing data versions, branches, merges, and commits. This can be very helpful when multiple data engineers are working with and transforming data at the same time. Nessie allows each engineer to work in separate branches while maintaining a single source of truth in the form of the main branch. This functionality empowers data engineering teams to maintain data quality collaboratively in the face of unrelenting change. This article provides a step-by-step guide demonstrating how Nessie, Dremio and MinIO work together to enhance data quality and collaboration in your data engineering workflows. Whether you're a data engineer, ML engineer, or just a modern data lake enthusiast, this blog equips you with the knowledge and tools needed to effectively enhance your data versioning practices. Understanding the Basics: Nessie Workloads Nessie allows for Git-like workloads that let you test, develop and push to production. Let's break down some key concepts: Just like in Git, branches in Nessie allow you to work on different data engineering tasks concurrently. For example, you might have a branch for feature development, data cleaning and data transformation. Each branch can have its own set of data changes. Branches: In Nessie, a commit represents a snapshot of the data at a specific point in time. When you make changes to your data, you create a new commit, which records those changes. Commits are linked to a specific branch. When you need to revert your data to a stable or known-good state, Nessie makes it easy to select a specific commit and roll back to that data version. This ensures data quality and consistency. Commits: : Nessie allows you to merge the changes from one branch into another. This is similar to merging code changes in Git. When you merge a branch, the data changes made in that branch become part of the target branch. Merges The Components is a distributed analytics engine that operates as an open-source platform, offering an intuitive self-service interface for data exploration, transformation and collaborative work. Its design is grounded in a high-speed columnar memory format. Dremio Apache Arrow, We’ve already explored how to deploy Dremio with Kubernetes, and also how to . For additional information about Dremio, consult . query Iceberg tables on MinIO with Dremio Dremio Resources is high-performance object storage. Renowned for its exceptional and , MinIO serves as a pivotal component in building and maintaining modern data lake infrastructures. MinIO empowers architects to efficiently manage and store massive volumes of data on-prem, on bare metal, on the edge, or on any of the public clouds. MinIO speed scalability is an open table format suited for managing large volumes of data in data lakes. Unique features like time travel, dynamic schema evolution, and Partition Evolution make it a game-changer, allowing query engines to work concurrently on the same data safely and efficiently. See for more information about Iceberg features. Apache Iceberg MinIO The Definitive Guide to Lakehouse Architecture with Iceberg and Getting Started You’ll need the and for this tutorial. The easiest way to get both if you don’t already have them is to install Docker Desktop. Docker Engine Docker Compose This part of the tutorial is based on Dremio’s . You can get the repo for this project . blog post here To begin, open a terminal and navigate to the folder where you cloned/downloaded the repo, and run the command below to start up Dremio. docker-compose up dremio Wait a few moments and then navigate to to access Dremio. Update the requested fields and then click Next. http://localhost:9047 Next, run the following command to start MinIO. docker-compose up minio The final command below will start up Nessie. docker-compose up docker-compose up nessie Navigate to to log in to MinIO with the username and password of . You’ll be prompted to create a bucket. http://localhost:9001 minioadmin:minioadmin Make one named . iceberg-datalake Then head back over to Dremio at and click on Add Source and select Nessie. http://localhost:9047 Set the Name to nessie Set the endpoint URL to http://nessie:19120/api/v2 Set the authentication to none Don’t click Save yet. Instead, in the navigation panel on the left, click Storage. MinIO is S3-API compatible object storage and can use the same connection pathways as AWS S3. For your access key, set minioadmin For your secret key, set minioadmin Set root path to /iceberg-datalake Scroll down for the next set of instructions. Click on the button under to create and configure the following properties. Add Property Connection Properties to fs.s3a.path.style.access true to fs.s3a.endpoint minio:9000 to dremio.s3.compat true Uncheck Encrypt connection Then click Save. You should now see Nessie Catalogs in your data sources. Create a Data Source In Dremio navigate to SQL Runner on the left-hand side. Make sure that in the upper right area of the text editor is set to our Nessie source. Otherwise, you will have to reference the context like instead of just to run this query. Copy and paste the SQL below and run. Context nessie.SalesData SalesData CREATE TABLE SalesData (
    id INT,
    product_name VARCHAR,
    sales_amount DECIMAL,
    transaction_date DATE
) PARTITION BY (transaction_date); Run the query below to insert data into the table you just created. INSERT INTO SalesData (id, product_name, sales_amount, transaction_date)
VALUES
    (1, 'ProductA', 1500.00, '2023-10-15'),
    (2, 'ProductB', 2000.00, '2023-10-15'),
    (3, 'ProductA', 1200.00, '2023-10-16'),
    (4, 'ProductC', 1800.00, '2023-10-16'),
    (5, 'ProductB', 2200.00, '2023-10-17'); Navigate back to MinIO to see that your data lake has been populated with the Iceberg tables. Branching and Merging with Nessie Return to Dremio at . Begin by querying a table on the main branch using the syntax: http://localhost:9047 AT BRANCH SELECT * FROM  nessie.SalesData AT BRANCH main; Create an ETL (Extract Transform and Load) branch to allow you to play around with and transform data without impacting production. CREATE BRANCH etl_06092023 in nessie Within the ETL branch, insert new data into the table: USE BRANCH etl_06092023 in nessie;
INSERT INTO nessie.SalesData (id, product_name, sales_amount, transaction_date) VALUES
(6, 'ProductC', 1400.00, '2023-10-18'); Confirm the immediate availability of the new data within the ETL branch: SELECT * FROM nessie.SalesData AT BRANCH etl_06092023; Note the isolation of changes from users on the main branch: SELECT * FROM nessie.SalesData AT BRANCH main; Merge the changes from the ETL branch back into the main branch: MERGE BRANCH etl_06092023 INTO main in nessie; Select the main branch again to see that the changes have indeed been merged. SELECT * FROM nessie.SalesData AT BRANCH main This branching strategy enables data engineers to independently handle numerous transactions across multiple tables. When they are ready, data engineers can merge these transactions into a single, comprehensive multi-table transaction within the main branch. Conclusion This blog post delved into the power of Git-like version control in data engineering, emphasizing how seamlessly manages data versions, branches and merges. This step-by-step guide demonstrates how Nessie, in collaboration with Dremio and MinIO, as the object storage foundation, enhances data quality and collaboration in data engineering workflows. Nessie Let us know what your data lakehouse looks like at or on . hello@minio.io our Slack channel Also appears . here

Data Lake Mysteries Revealed: Nessie, Dremio, and MinIO Make Waves

ডেটা লেক রহস্য প্রকাশিত হয়েছে: নেসি, ড্রেমিও এবং মিনিও মেক ওয়েভস

Data-Lake-Rätsel aufgedeckt: Nessie, Dremio und MinIO sorgen für Aufsehen

Misterios del lago de datos revelados: Nessie, Dremio y MinIO hacen olas

Les mystères du Data Lake révélés : Nessie, Dremio et MinIO font des vagues

डेटा लेक रहस्य का खुलासा: नेस्सी, ड्रेमियो और मिनिओ मेक वेव्स

データレイクの謎が明らかに: Nessie、Dremio、MinIO が波紋を起こす

데이터 레이크의 미스터리 공개: Nessie, Dremio 및 MinIO가 물결을 일으키다

Mistérios do Data Lake revelados: Nessie, Dremio e MinIO fazem ondas

Тайны озера данных раскрыты: Несси, Дремио и MinIO вызывают фурор

Data Lake'in Gizemleri Ortaya Çıktı: Nessie, Dremio ve MinIO Dalga Yaratıyor

Những bí ẩn về hồ dữ liệu được tiết lộ: Nessie, Dremio và MinIO tạo nên làn sóng

数据湖谜团揭开：Nessie、Dremio 和 MinIO 掀起波澜

Amid the fervor to adopt AI is a critical and often overlooked truth -  the success of any AI initiative is intrinsically tied to the quality, reliability and performance of the underlying data infrastructure. If you don't have the proper foundation, you are limited in what you can build and therefore what you can achieve. Your data infrastructure is the bedrock upon which your entire AI infrastructure is built. It's where data is collected, stored, processed, and transformed. Training models using supervised, unsupervised, and reinforcement learning require storage solutions that can handle structured data - like a Data Warehouse. On the other hand, if you are training Large Language Models (LLMs), you must manage unstructured data - documents in their raw and processed form. A Modern Datalake, or Lakehouse, is the foundation for both these different flavors of AI. A Modern Datalake is one-half Data Warehouse and one-half Data Lake and uses object storage for everything. More recently, we have seen the rise of open table formats. Open Table Formats (OTFs) like Apache Iceberg, Apache Hudi, and Delta Lake make it seamless for object storage to be used within a data warehouse. The rest of this article will examine how to leverage the characteristics of the Modern Datalake that differentiate it from conventional solutions like proprietary Data Warehouses and appliances. To build a foundation for AI infrastructure you need the following: Disaggregation of Compute and Storage Scale Out (Not Up) Software Defined Cloud Native Commodity Hardware If we agree on the above, a series of best practices emerge that focus on two areas of performance. If incorporated, the modern Datalake will be both fast and scalable. These best practices include: Optimizing Drive Price and Performance Incorporate a High-Speed Network Disaggregating Compute and Storage Disaggregating compute and storage within your data infrastructure means that separate resources are used for compute and storage. This contrasts conventional storage solutions, where everything comes packaged in a single server or, worse yet, an appliance. However, Modern Datalakes take disaggregation to another level. If the Data Lake and the Data Warehouse had completely different storage requirements, we could use two separate instances of an object store, as shown below. Additionally, If the Data Warehouse needs to support workloads that require conflicting configurations, then you could use more than one processing engine. This is shown below. A composable infrastructure allows you to scale your compute and storage resources independently. This means you can allocate more resources to the part of your infrastructure that needs it most rather than upgrading both compute and storage together. It results in cost-effective scaling as you only invest in the required resources. Scale-Out Not Up AI workloads are data intensive, are often distributed across multiple CPUs or GPUs, use copious compute power for training, and necessitate real-time inference. Scaling out, not up, helps optimize performance and accommodate high-speed networks. Scaling out and scaling up are two different approaches to increasing the capacity and performance of your data infrastructure. However, scaling out is proving to be the more viable approach as advances are made in clustering platforms like Kubernetes and more and more solutions strive to be cloud native. Scaling out in a disaggregated infrastructure provides: - If one node is busy, another node can take on a new request, reducing wait times and increasing throughput. If one node fails, the workload can be shifted to other nodes, reducing downtime and ensuring continuity. High Availability and Fault Tolerance - Scaling out can provide better performance by distributing workloads across multiple nodes or servers to handle larger volumes of data and more concurrent requests. Scaling out is also more flexible because you can add or remove nodes as needed, making it easier to adjust to fluctuating workloads or to accommodate seasonal variations. Performance and Flexibility - Maintenance and upgrades are simplified when you scale out. Instead of taking a critical system offline for upgrades, you can perform maintenance on individual storage or compute nodes without disrupting the entire infrastructure. Operationally and Resource Efficient Cloud Native + Software Defined The last component of leveraging the Modern Datalake to build a strong foundation for AI is taking a cloud-native, software-defined approach. Containers like Docker and container orchestration tools, like Kubernetes, make cloud-native architectures possible. All the components of a Modern Datalake run in containers that run in Kubernetes. Therefore, a Modern Datalake is cloud native. "Software-defined" refers to an approach in which software controls and manages the configuration, functionality, and behavior of hardware components, often in the context of computer systems and networking. This is the building block of the infrastructure as code movement where the emphasis is on smart software and dumb fast hardware. Software-defined storage abstracts and manages the storage resources through software, making allocating and managing storage capacity across different devices and storage media easier. Built for Speed: NVMe and 100GbE To take full advantage of your commodity hardware and software-defined architecture - you need two more key pieces. The first is NVMe drives. Modern, performance oriented workloads, the random nature of read/write, the rise of small objects and falling SSD pricing all favor a . Do that math, the upfront may be higher, the TCO will be lower. NVMe centric architecture The second component is 100GbE networking. In a software-defined world, the network turns out to be the bottleneck in many setups even at 100GbE. Here are some of those scenarios: - AI workloads often process massive datasets, such as images, videos, natural language text, and sensor data. High-speed networks can quickly transfer these large datasets between storage and processing units, reducing data transfer bottlenecks. Data Intensive - Many AI tasks involve distributed computing across multiple CPUs or GPUs. High-speed networks enable efficient communication and data exchange between these devices, ensuring the computing clusters work in parallel effectively. Distributed Computing - Training deep learning models, especially LLMs like transformers or convolutional neural networks, requires a lot of data and computational power. A high-speed network allows faster data loading and synchronization between distributed GPUs, which can significantly speed up training times. Model Training - Low-latency and high-throughput networks are essential for responsive applications that incorporate AI. A high-speed network ensures minimal delay between a user request and a response from a model. Real-time Inference Foundational Concepts By adhering to these principles: the disaggregation of compute and storage, scale-out, not up, dumb, fast hardware and smart cloud native software the enterprise can construct a Modern Datalake that has the right foundation for meeting these requirements and pushing your AI initiatives forward. You cannot construct a building on a poor foundation, just ask the ancient Egyptians. The AI game is about performance at scale, and this requires the right foundation. To skimp on the foundation is to accumulate technical debt that will topple your Jenga tower after a few minutes. Build smart, put the foundation in place.

The Principles to Keep In Mind When Building a Modern Datalake for Your AI Infrastructure

আপনার AI পরিকাঠামোর জন্য একটি আধুনিক ডেটালেক তৈরি করার সময় যে নীতিগুলি মাথায় রাখতে হবে

Die Grundsätze, die Sie beim Aufbau eines modernen Datalake für Ihre KI-Infrastruktur beachten sollten

Los principios a tener en cuenta al construir un lago de datos moderno para su infraestructura de IA

Les principes à garder à l'esprit lors de la création d'un datalake moderne pour votre infrastructure d'IA

आपके एआई इन्फ्रास्ट्रक्चर के लिए आधुनिक डेटालेक का निर्माण करते समय ध्यान में रखने योग्य सिद्धांत

AI インフラストラクチャ用の最新のデータレイクを構築する際に留意すべき原則

AI 인프라를 위한 최신 데이터레이크를 구축할 때 명심해야 할 원칙

Os princípios a serem considerados ao construir um Datalake moderno para sua infraestrutura de IA

Принципы, которые следует учитывать при создании современного озера данных для вашей инфраструктуры искусственного интеллекта

Yapay Zeka Altyapınız için Modern Bir Datalake Oluştururken Akılda Tutulması Gereken İlkeler

Những nguyên tắc cần ghi nhớ khi xây dựng Datalake hiện đại cho cơ sở hạ tầng AI của bạn

为人工智能基础设施构建现代数据湖时要牢记的原则

As much as we say is an all-in-one data platform that is capable of various analytics workloads, it is always compelling to demonstrate that by real use cases. That's why I would like to share this user story with you. It is about how they leverage the capabilities of Apache Doris in reporting, customer tagging, and data lake analytics and achieve high performance. Apache Doris This fintech service provider is a long-term user of Apache Doris. They have almost 10 clusters for production, hundreds of Doris backend nodes, and thousands of CPU Cores. The total data size is near 1 PB. Every day, they have hundreds of workflows running simultaneously, receive almost 10 billion new data records, and respond to millions of data queries. Before migrating to Apache Doris, they used ClickHouse, MySQL, and Elasticsearch. Then, frictions arise from their ever-enlarging data size. They found it hard to scale out the ClickHouse clusters because there were too many dependencies. As for MySQL, they had to switch between various MySQL instances because one MySQL instance had its limits, and cross-instance queries were not supported. Reporting From ClickHouse + MySQL to Apache Doris Data reporting is one of the major services they provide to their customers and they are bound by an SLA. They used to support such service with a combination of ClickHouse and MySQL, but they found significant fluctuations in their data synchronization duration, making it hard for them to meet the service levels outlined in their SLA. Diagnosis showed that it was because the multiple components add to the complexity and instability of data synchronization tasks. To fix that, they have used Apache Doris as a unified analytic engine to support data reporting. Performance improvements With Apache Doris, they ingest data via the method and reach an SLA compliance rate of over 99% in terms of data synchronization performance. Broker Load As for data queries, the Doris-based architecture maintains an of less than and a of less than . This is a 50% speedup compared to the old architecture. average query response time 10s P90 response time 30s Tagging Tagging is a common operation in customer analytics. You assign labels to customers based on their behaviors and characteristics so that you can divide them into groups and figure out targeted marketing strategies for each group of them. In the old processing architecture, where Elasticsearch was the processing engine, raw data was ingested and tagged properly. Then, it will be merged into JSON files and imported into Elasticsearch, which provides data services for analysts and marketers. In this process, the merging step was to reduce updates and relieve load for Elasticsearch, but it turned out to be a troublemaker: Any problematic data in any of the tags could spoil the entire merging operation and thus interrupt the data services. The merging operation was implemented based on Spark and MapReduce and took up to 4 hours. Such a long time frame could encroach on marketing opportunities and lead to unseen losses. Then Apache Doris takes this over. Apache Doris arranges tag data with its data models, which process data fast and smoothly. The aforementioned merging step can be done by the , which aggregates tag data based on the specified Aggregate Key upon data ingestion. The is handy for partial column updates. Again, all you need is to specify the Unique Key. This enables swift and flexible data updating and saves you from the trouble of replacing the entire flat table. You can also put your detailed data into a to speed up certain queries. . Aggregate Key model Unique Key model Duplicate model In practice, it took the user 1 hour to finish the data ingestion, compared to 4 hours with the old architecture In terms of query performance, Doris is equipped with well-developed bitmap indexes and techniques tailored to high-concurrency queries, so in this case, it can finish and reach over . customer segmentation within seconds 700 QPS in user-facing queries Data lake analytics In data lake scenarios, the data size you need to handle tends to be huge, but the data processing volume in each query tends to vary. To ensure fast data ingestion and high query performance of huge data sets, you need more resources. On the other hand, during non-peak time, you want to scale down your cluster for more efficient resource management. How do you handle this dilemma? Apache Doris has a few features that are designed for data lake analytics, including Multi-Catalog and Compute Node. The former shields you from the headache of data ingestion in data lake analytics while the latter enables elastic cluster scaling. The mechanism allows you to connect Doris to a variety of external data sources so you can use Doris as a unified query gateway without worrying about bulky data ingestion into Doris. Multi-Catalog The of Apache Doris is a backend role that is designed for remote federated query workloads, like those in data lake analytics. Normal Doris backend nodes are responsible for both SQL query execution and data management, while the Compute Nodes in Doris, as the name implies, only perform computation. Compute Nodes are stateless, making them elastic enough for cluster scaling. Compute Node The user introduces Compute Nodes into their cluster and deploys them with other components in a hybrid configuration. As a result, the cluster automatically scales down during the night, when there are fewer query requests, and scales out during the daytime to handle the massive query workload. This is more resource-efficient. For easier deployment, they have also optimized their Deploy on Yarn process via Skein. As is shown below, they define the number of Compute nodes and the required resources in the YAML file, and then pack the installation file, configuration file, and startup script into the distributed file system. In this way, they can start or stop the entire cluster of over 100 nodes within minutes using one simple line of code. Conclusion For data reporting and customer tagging, Apache Doris smoothens data ingestion and merging steps and delivers high query performance based on its own design and functionality. For data lake analytics, the user improves resource efficiency by elastic scaling of clusters using the Compute Node. Along their journey with Apache Doris, they have also developed a data ingestion task prioritizing mechanism and contributed it to the Doris project. A gesture to facilitate their use case ends up benefiting the whole . This is a great example of open-source products thriving on user involvement. open source community Also published . here

Data Analytics: Apache Doris' Impact in Reporting, Tagging, and Data Lake Operations

T3Go is China’s first platform for smart travel based on the Internet of Vehicles. In this article, Trevor Zhang and Vino Yang from T3Go describe the evolution of their data lake architecture, built on cloud-native or open-source technologies, including Alibaba OSS, Apache Hudi, and Alluxio. Today, their data lake stores petabytes of data, supporting hundreds of pipelines and tens of thousands of tasks daily. It is essential for business units at T3Go, including Data Warehouse, Internet of Vehicles, Order Dispatching, Machine Learning, and self-service query analysis. In this blog, you will see how we slashed data ingestion time by half using Hudi and Alluxio. Furthermore, data analysts using Presto, Hudi, and Alluxio saw the queries speed up by 10 times. We built our data lake based on data orchestration for multiple stages of our data pipeline, including ingestion and analytics. T3Go data lake Overview Prior to the data lake, different business units within T3Go managed their own data processing solutions, utilizing different storage systems, ETL tools, and data processing frameworks. Data for each became siloed from every other unit, significantly increasing cost and complexity. Due to the rapid business expansion of T3Go, this inefficiency became our engineering bottleneck. We moved to a unified data lake solution based on Alibaba OSS, an object store similar to AWS S3, to provide a centralized location to store structured and unstructured data, following the design principles of ; all the applications access OSS storage as the source of truth, as opposed to different data silos. This architecture allows us to store the data as-is without having to first structure the data and run different types of analytics to guide better decisions, building dashboards and visualizations from big data processing, real-time analytics, and machine learning. Multi-cluster Shared-data Architecture Efficient Near Real-time Analytics Using Hudi Our business in smart travel drives the need to process and analyze data in a near real-time manner. With a traditional data warehouse, we faced the following challenges: High overhead when updating due to long-tail latency. High cost of order analysis due to the long window of a business session Reduced query accuracy due to late or ad-hoc updates Unreliability in data ingestion pipeline. Data lost in the distributed data pipeline cannot be reconciled. High latency to access data storage. As a result, we adopted Apache Hudi on top of OSS to address these issues. The following diagram outlines the architecture: Enable near real-time data ingestion and analysis With Hudi, our data lake supports multiple data sources, including Kafka, MySQL binlog, GIS, and other business logs in near real-time. As a result, more than 60% of the company’s data is stored in the data lake, and this proportion continues to increase. We are also able to speed up the data ingestion time down to a few minutes by introducing Apache Hudi into the data pipeline. Combined with big data interactive query and analysis frameworks, such as Presto and SparkSQL, real-time data analysis and insights are achieved. Enable Incremental processing pipeline With the help of Hudi, it is possible to provide incremental changes to the downstream derived table when the upstream table updates frequently. Even with a large number of interdependent tables, we can quickly run partial data updates. This also effectively avoids updating the full partitions of cold tables in the traditional Hive data warehouse. Accessing Data using Hudi as a unified format Traditional data warehouses often deploy Hadoop to store data and provide batch analysis. Kafka is used separately to distribute Hadoop data to other data processing frameworks, resulting in duplicated data. Hudi helps effectively solve this problem; we always use Spark pipelines to insert new updates into the Hudi tables, then incrementally read the update of Hudi tables. In other words, Hudi tables are used as the unified storage format to access data. Efficient Data Caching Using Alluxio In the early version of our data lake without Alluxio, data received from Kafka in real-time is processed by Spark and then written to OSS data lake using Hudi DeltaStreamer tasks. With this architecture, Spark often suffered high network latency when writing to OSS directly. Since all data is in OSS storage, OLAP queries on Hudi data may also be slow due to a lack of data locality. To address the latency issue, we deployed Alluxio as a data orchestration layer, co-located with computing engines such as Spark and Presto, and used Alluxio to accelerate read and write on the data lake as shown in the following diagram: Data in formats such as Hudi, Parquet, ORC, and JSON are stored mostly on OSS, consisting of 95% of the data. Computing engines such as Flink, Spark, Kylin, and Presto are deployed in isolated clusters, respectively. When each engine accesses OSS, Alluxio acts as a virtual distributed storage system to accelerate data, being co-located with each of the computing clusters. Specifically, here are a few applications leveraging Alluxio in the T3Go data lake. Data lake ingestion We mount the corresponding OSS path to the Alluxio file system and set Hudi’s parameter value to use the scheme in place of scheme. Spark pipelines with Hudi continuously ingest data to Alluxio. After data is written to Alluxio, it is asynchronously persisted from the Alluxio cache to the remote OSS every minute. These modifications allow Spark to write to a local Alluxio node instead of writing to remote OSS, significantly reducing the time for the data to be available in the data lake after ingestion. target-base-path alluxio:// oss:// Data analysis on the lake We use Presto as an ad-hoc query engine to analyze the Hudi tables in the lake, co-locating Alluxio workers on each Presto worker node. When Presto and Alluxio services are co-located and running, Alluxio caches the input data locally in the Presto worker, which greatly benefits Presto for subsequent retrievals. On a cache hit, Presto can read from the local Alluxio worker storage at memory speed without any additional data transfer over the network. Concurrent accesses across multiple storage systems In order to ensure the accuracy of training samples, our machine learning team often synchronizes desensitized data in production to an offline machine learning environment. During synchronization, the data flows across multiple file systems, from production OSS to an offline HDFS followed by another offline Machine Learning HDFS. This data migration process is not only inefficient but also error-prone for modelers because multiple different storages with varying configurations are involved. Alluxio helps in this specific scenario by mounting the destination storage systems under the same filesystem to be accessed by their corresponding logical paths in Alluxio namespace. By decoupling the physical storage, allows applications with different APIs to access and transfer data seamlessly. This data access layout also improves performance. Microbenchmark Overall, we observed the following improvements with Alluxio: It supports a hierarchical and transparent caching mechanism. It supports cache promote mode when reading. It supports asynchronous writing mode. It supports LRU recycling strategy. It has pin and TTL features After comparison and verification, we choose to use Spark SQL as the query engine. Our performance testing queries the Hudi table, comparing Alluxio + OSS together against OSS directly as well as HDFS. In the stress test shown above, after the data volume is greater than a certain magnitude (2400W), the query speed using Alluxio+OSS surpasses the HDFS query speed of the hybrid deployment. After the data volume is greater than 1E, the query speed starts to double. After reaching 6E data, it is up to 12 times higher than querying native OSS and 8 times higher than querying native HDFS. The improvement depends on the machine configuration. Based on our performance benchmarking, we found that the performance can be improved by over 10 times with the help of Alluxio. Furthermore, the larger the data scale, the more prominent the performance improvement. Next Step As T3Go’s data lake ecosystem expands, we will continue facing the critical scenario of compute and storage segregation. With T3Go’s growing data processing needs, our team plans to deploy Alluxio on a larger scale to accelerate our data lake storage. In addition to the deployment of Alluxio on the data lake computing engine, which currently is mainly SparkSQL, we plan to add a layer of Alluxio to the OLAP cluster using Apache Kylin and an ad_hoc cluster using Presto. The goal is to have Alluxio cover all computing scenarios, with Alluxio interconnected between each scene to improve the read and write efficiency of the data lake and the surrounding lake ecology. Conclusion As mentioned earlier, Hudi and Alluxio cover all scenarios of Hudi’s near real-time ingestion, near real-time analysis, incremental processing, and data distribution on DFS, among many others, and play the role of a powerful accelerator on data ingestion and data analysis on the lake. With Hudi and Alluxio together, our R&D engineers shortened the time for data ingestion into the lake by up to a factor of 2. Data analysts using Presto, Hudi, and Alluxio in conjunction to query data on the lake saw their queries speed up by 10 times faster. Furthermore, the larger the data scale, the more prominent the performance improvement. Alluxio is an important part of T3Go’s plan to become a leading enterprise data lake in China. We look forward to seeing further integration with Alluxio in T3Go’s data lake ecosystem.

Building High-Performance Data Lake Using Apache Hudi and Alluxio 

Data lakes provide a myriad of benefits. They are data agnostic and don’t require you to define a schema upfront. However, without a proper structure, it may be challenging to find the data that you need. In this article, we’ll address this problem by creating a FastAPI abstraction allowing us to query the AWS Glue metadata catalog. Why FastAPI for a data discovery service? Photo by nine koepfer If we would implement it as a Python package instead, we would assume that everyone knows enough Python to use it. However, your company may have other developers (Java, C#, JavaScript, Go, …) that need access to data from a data lake. By building a REST-ful service for data discovery, you are providing a interface for everyone. This approach has further : programming-language agnostic API-first benefits If you later decide that you want to build a for business users, your task will become much easier since you already have a REST API that provides data as JSON. UI You can combine all the functionality for this purpose in a . single service If you need to add new methods or change the existing ones, you can deploy easier by creating a (ie, api_url/v3) without breaking any workloads that depend on the current version. changes new API version You can to include real-time data streams, unstructured data, as well as to allow querying access logs to identify which data is used most frequently and by which users. extend the functionality We can start by listing all the methods that can be potentially useful for data discovery: listing all databases or , schemas listing all for a specific schema, tables filtering for table names with a specific or , prefix suffix doing a on table names (say, all tables that have anything to do with orders or customers), full-text search doing a on table comments (ex., if we want to find any column related to ), full-text search logistics showing a ( ), full table definition all columns, data types, and column comments showing for a specific table, partitions showing from S3 below a specific partition, all files showing all columns with specific , comment showing all tables that contain a specific , column name finally, the best way for data discovery is to just of a full table. look at the first X rows We have three choices: either using lower-level AWS SDKs, doing queries on Athena’s information_schema, or leveraging awswrangler. In the end, we may combine all of them to satisfy the requirements from the previous section. #1. To list all databases ( ) and tables, we can use the package, especially the module wr.catalog. i.e., schemas awswrangler #2. To filter for table names with a specific or , we can also use wr.catalog.tables. The same is true for retrieving and doing a on those definitions: prefix suffix table definition full-text search #3. List all of a specific table: partitions #4. Show table’s DDL: #5. directory or show specific objects according to file type ( ) and last modified date: List S3 path ex. parquet files #6. Query to find the dataset that you need: column comments #7. Describe Athena : table definition #8. Search for specific : column names The above methods constitute a basic MVP for a data discovery service. It allows querying schemas, tables, columns, column comments ( ), showing a preview of the data, as well as exploring the underlying files, directories, and partitions. aka data dictionary By leveraging services such as AWS X-Ray and Lake Formation, we could add methods to query usage access patterns and identify the most and the least used datasets. If you don’t want to attach your boto3_session separately in each API method, you can attach a global session using: Side note: Implementation: FastAPI MVP The full code for this MVP can be found in . Let’s briefly discuss some of the details. this Github repository In the following demo, we can see all the endpoints applied on a . Brazilian E-commerce dataset from Kaggle The screencast above demonstrates how we can use this API to discover and query e-commerce data stored in an S3 data lake. The end users can identify tables or schemas that they need. By drilling down into the underlying files, data scientists and analysts can explore this data in an interactive notebook or other tools of their choice. Reading a specific parquet, CSV, Excel, or JSON file from S3 with awswrangler is as simple as: How can we serve this API at scale? There are several options to deploy a REST API. If you already use AWS, you may find it useful to leverage serverless services built specifically for building resilient and scalable APIs: and . AWS Lambda API Gateway With the excellent package called mangum, converting our FastAPI to a Lambda handler is as simple as importing this package ( ) and adding a single line of code: handler = Mangum(app=app). from mangum import Mangum The Dockerfile to deploy the API to AWS Lambda looks as follows: We use the official Python Lambda image with Python 3.8 and all required . Then, we copy our remaining API code and specify the that will be used as an entry point to our container. install packages Lambda handler Finally, we have to push the container to ECR — AWS container registry. To replicate this demo, replace 123456789 with your AWS account ID, and adjust your AWS region name. In case you’re wondering: dda is my abbreviation for data-discovery-api. AWS Lambda The container image is deployed to ECR. Now, we need to . We choose the container image option and select our ECR image. create a Lambda function Since querying some tables can take longer than Lambda’s default of just three seconds, we can extend it to three minutes. timeout Finally, we need to attach IAM policies to grant our Lambda function permissions to retrieve data from AWS Glue, Athena, and S3 ( ). of course, we should use more granular permissions for production API Gateway After configuring the Lambda function with our container image, we can create a new API in the API Gateway console. We need to choose REST API and specify a name. Then, we can add a method and resource to configure Lambda Proxy integration. Once all that is set up, we can deploy the API and start serving it. To see it step by step, here is a screencast that demonstrates API Gateway configuration, deployment, and testing the API: How can we monitor our data discovery service? Imagine that you deployed a similar service at your company and it turned out to be a resounding success. Now you need to ensure that it stays , can be easily , and runs . At this point, you can either: healthy maintained without errors Keep an eye on it manually ( ), not recommended Build an yourself by using AWS CloudWatch and related services, observability dashboard Make use of observability tools The implications of each option: With option #1, you will likely go mad. With option #2, you would have to spend extra time building CloudWatch dashboards, jump between several CloudWatch logs and multiple AWS console tabs to fix any potential bugs. With option #3, you only have to sign in to your account  to scans all serverless resources and provides you with visualizations you need to track the health of your APIs and microservices. You can then optionally ( ) build a by combining API Gateway and Lambda resources, and on failure. highly recommended! project dashboard set alerts In the image above, we can see how Dashbird pointed out the task which timed out after 3 seconds ( ). It also reveals when the function experienced a cold start and shows the duration for each API call. default Lambda timeout Lambda For , we can see the average latency, number of API requests, and different types of errors that we encountered in the initial testing of the API. API Gateway Conclusion In this article, we investigated how we can build a REST-ful service for data discovery. By leveraging Python libraries such as FastAPI, awswrangler, boto3, and Mangum, we can build useful APIs in just a few hours rather than weeks. Additionally, by deploying this service to API Gateway and Lambda, we can serve this API at scale with no operational headaches. Lastly, by leveraging Dashbird, you can add observability to your serverless resources without having to install any CloudWatch agents, pulling logs, or building any dashboards. Previously published at https://dashbird.io/blog/data-discovery-api-aws-data-lake/

How To Use a FastAPI Abstraction To Query AWS Data Lake

Databases store data in a structured form. The structure makes it possible to find and edit data. With their structured structure, databases are used for data management, data storage, data evaluation, and targeted processing of data. In this sense, data is all information that is to be saved and later reused in various contexts. These can be date and time values, texts, addresses, numbers, but also pictures. The data should be able to be evaluated and processed later. The amount of data the database could store is limited, so enterprise companies tend to use data warehouses, which are versions for huge streams of data. Source A data warehouse (DWH) is a prepared, structured (tabular) recorded data source. In the simplest case, an SQL database that contains at least one data record. At the enterprise level, such a warehouse naturally quickly takes on larger dimensions, so that there are entire business intelligence departments that only deal with the business warehouse. A data lake records both structured and unstructured data and store it in various processing steps (raw, processed, analyzed).The high possible variance of data sources (e.g. images, videos, text) and the raw data allows data scientists to carry out further analyzes that are not possible in a DWH. In abstract terms, the data warehouse can be considered as an Excel file on the computer, while the data lake represents a file folder. Like an Excel file, the DWH contains very structured data with named columns in a fixed schema. Adding new entries is not a problem, new columns are more difficult depending on the existing content. Once the data has been typed in and saved, the originals can no longer be found, so the file must be relied on. The content, in turn, is very easy to use for certain applications: visualizations or simple arithmetic operations for KPI calculation are very convenient. If you now take the data lake in the same example, it behaves like a file folder on the hard drive. You can store a large number of (original) data without having to format, type or structure them beforehand. However, if you want to continue working with the data, you have to prepare the data first and you cannot simply pull sums across columns. If this processing has taken place, however, the data can then also be saved in the folder again. Even many Excel files — i.e. data warehouses — can thus be generated and saved in the folder for further processing. Conclusion Hopefully, this simple example explains the difference between a database, data warehouse, and data lake. For companies, depending on the maturity of the data-driven business, it makes sense to use one or both of the infrastructures. A DWH allows a wide range of users quick access to structured data for analysis, while a data lake enables advanced users, for example, data engineers and data scientists, to apply machine learning and other advanced analytics methods. However, if you do need a deeper explanation of the differences between these 3 terms and a deep dive into Data Lake functioning ; read my previous article .

Database Vs Data Warehouse Vs Data Lake: A Simple Explanation

This is a hands-on course on how to deploy a fully Serverless web app using the . You will learn how to: AWS CDK Structure **CDK Stacks **to deploy an application from end-to-end Deploy a integrated with for dynamic requests processing Store data in a fast and cost-effective way with REST API AWS Lambda DynamoDB Use as a source for in an event-driven architecture DynamoDB streams Lambda Ingest and manipulate loads of data streams with Kinesis Firehose Deploy and query a with , and your entire application health in a single place using Data Lake Athena S3 GlueMonitor Dashbird You can use the resources declared in this demo application as a starting point to mix and adapt to your own architectures later, which should save you quite some time. The App and Architecture The demo app is a public blog where anyone can read, publish and like posts. It’s available . Go ahead and publish something in the top-left corner (yellow button) and also “like” articles already published. Check out the codebase on . on this link this repo Frontend Backend Data Lake and Analytical Querying What is the AWS CDK? CDK stands for Cloud Development Kit. Think of it as CloudFormation (CF) in your preferred language (Python, Typescript, C#, etc). Roughly speaking, it works like this: 1. You declare cloud resources using classes provided by the CDK libraries. Example: aws_cdk aws_s3
my_bucket = aws_s3.Bucket(self, ) from import 'MyBucket' 2. Run CDK deploy 3. CDK translates this to a CloudFormation template and deploy it on AWS for you. In case you would like to dig deeper, that will get your basics started. I also strongly recommend reading the . AWS also has a workshop official CDK documentation Advantages using the AWS CDK Use languages that are more expressive than YAML or JSON, for instance. Less - much less! - verbose than CloudFormation templates. Easier to apply reusability and inheritance principles to infrastructure code. Better integration with IDEs for code completion, IntelliSense, etc. Possible to test your infra code, just as any other software Portable: since it’s just a wrapper around CF, we can easily port it to JSON or YAML Disadvantages of using the AWS CDK Although released as a stable project by the AWS team, many parts (a lot of the good ones) are still experimental and APIs may change in backwards incompatible ways. It’s under constant development. During the preparation of this course, I had to upgrade my libraries three times. Documentation is still lacking at some parts and you will need to look at the CDK code, occasionally, to understand how to declare certain things. Deploy it yourself Although we have provided an online demo, you can also deploy this app in your own AWS account: Clone the repo: git clone git@github.com:byrro/serverless-website-demo.git sls-demo; cd sls-demo Setup your virtual environment: virtualenv -p /urs/bin/python3.8 .env; source .env/bin/activate; pip install -r requirements Specify an AWS account ID: ** export AWS_ACCOUNT_ID=1234567890 Deploy all three stacks: cdk deploy sls-blog; cdk deploy sls-blog-api; cdk deploy sls-blog-analytical ** You can also hard code your Account ID in the CDK project, as I’ll show in a minute; When starting a new project from scratch, you would run . This is not necessary for this demo, since the project is already created. cdk init --language [python|typescript|...] Monitoring Deploying this architecture in the cloud and blindly believing it will work flawlessly is not reasonable. We want to be the first one to know when something is not right to act upon it as quickly as possible. In this project, I used for its ease of use and seamless integration. Instead of having to deploy an agent inside my code, Dashbird plugs into my Stacks that I can deploy with the effort of one click. It not only monitors Lambda function errors, but also other resources that we’re using, such as DynamoDB tables. They even suggest cross-referenced against industry best practices. Dashbird through a CloudFormation template insights for architectural improvements Finally, Dashbird offers a free-forever plan. It’s a no-brainer to try it out by . registering for free How a CDK project is structured A CDK project creates an “Application”. This app may have one or more “Stacks”. A Stack is a group of cloud resources (Lambda functions, S3 buckets, etc) that are instantiated using CDK classes. It’s also possible to have multiple applications in a single CDK project. App Object Creating a CDK app is as simple as: app = core.App() When you run , an initial application with basic boilerplate code is created for you in the project root, under . cdk init --language [language] app.py The next thing we need is an environment, which is composed of an AWS Account ID and Region: env = env = core.Environment(
    account= ,
    region= ,
) 1234567890 'us-east-1' Declaring an environment is not required (CDK can infer from your AWS credentials), but is a good practice. Most of us work with multiple AWS accounts. It’s easy to mess around with several projects, accounts, credentials. When we explicitly set the environment in the CDK app, it’s locked and prevents mistaken deployments. Now we declare our stacks: my_project.my_project_stack MyStack

my_stack = MyStack(
    app, ,
    env=env,
) from import 'my-stack' This is how we our stacks for deployment. In the next section we’ll see how to those stacks. instantiate declare Stack Object The Stack object is where we our AWS resources. It inherits from the core.Stack CDK class and accepts a scope - which is our app object - a string identifier and an environment. declare ( . ): ( , : . , : , : . ,
            ** ,
            ) -> : (). ( , , ** )

        # class MyStack core Stack def __init__ self scope core Construct id str env core Environment kwargs None super __init__ scope id kwargs Declare AWS resources here Declaring AWS resources To declare AWS resources, we need a specific library for each service. Here’s a list of all and their counterparts. Other flavors are and . Python libraries Typescript Java .NET Let’s see how a basic REST API would be declared (typing expressions were removed for readability purposes): aws_cdk aws_apigateway, aws_lambda = aws_lambda.Function(
            self, ,
            runtime=aws_lambda.Runtime.PYTHON_3_8,
            code=aws_lambda.Code.asset( my_lambda.handler sls-blog-rest-api-gateway from import ( . ): ( , , , ): (). ( , , ** ) class MyStack core Stack def __init__ self scope id env super __init__ scope id kwargs my_lambda 'MyLambda' 'my_lambda_folder),
            handler=' ',
        )

        aws_apigateway.LambdaRestApi(
            self,
            ' ',
            handler=my_lambda,
        ) We first declare a Lambda function . We point its code to the . Inside this folder, there should be a file, containing a function called . This handler function should accept Lambda invocations normally (an and objects). my_lambda my_lambda_folder my_lambda.py handler event context Next, a is declared, using as the handler (not to confuse with the Lambda’s handler function). This will create a new API Gateway REST API integrated with using an . All HTTP requests will be routed to the Lambda function. LambdaRestApi my_lambda my_lambda AWS_PROXY integration type Our Project App & Stacks comprises one application with three Stacks. They’re all declared in the and files. This project app.py sls_website_stack.py Below we’ll walk through all Stacks in a high level. I encourage you to inspect the to learn how these resources are declared and also integrated. For example: a Kinesis Firehose is created in one Stack and referenced in another to include its name as an environment variable for the Lambda function that will interact with it. stacks file Except for the frontend static Stack - which is small - you will notice that resources are initialized with a (null) value in the beginning. The reason is that, even though the CDK is generally more succinct than CloudFormation, it can still be lengthy enough to clutter the view of the entire Stack. Having each resource declared first in one line, I can provide a short summary of everything that’s in the Stack and then instantiate the CDK classes in other methods. None = static_stack

        # SQS Queues
        self.queue_ddb_streams_dlq = None  # Dead-letter-queue DDB streams

        # DynamoDB Tables
        self.ddb_table_blog = None  # Single-table all blog content

        # DynamoDB Event Sources
        self.ddb_source_blog = None  # Blog table streams source

        # DynamoDB Indexes
        self.ddb_gsi_latest = None  # GSI ordering articles by timestamp

        # Lambda Functions
        self.lambda_blog = None  # Serves requests to the blog public API
        self.lambda_stream_reader = None  # Processes DynamoDB streams

        # Continues other resources... ( . ): ( , , , , ): (). ( , , ** ) . class SlsBlogApiStack core Stack def __init__ self scope id env static_stack super __init__ scope id kwargs self static_stack for for with Notice it takes another Stack object ( ) as an argument to its initialization. In the file, you can see that the is initialized passing the as an argument. static_stack app.py SlsBlogApiStack SlsBlogStack We use it to reference the CloudFront distribution domain ( ) in the Lambda environment variables. This variable can be used to customize the HTTP response header to comply with . This illustrates one way to easily integrate and reference information from one Stack into another within a CDK project. d1qmte5oc6ndq5.cloudfront.net Access-Control-Allow-Origin CORS standards At the end of the initialization, another method is called to instantiate the CDK classes for each resource and configure their parameters. self.create_cdk_resources() Next we’ll walk through each of our project’s Stacks. Stack 1: static website Our focus is on the Serverless backend, so the frontend here is terribly rough and simple. It’s stored in an S3 Bucket and distributed through a CloudFront CDN. CDK has a helpful class called BucketDeployment. It takes the contents of a directory and sync to an S3 bucket. In this case, we stored the frontend code in the folder. website_static aws_s3_deployment.BucketDeployment(
    self, ,
    sources=[aws_s3_deployment.Source.asset( )],
    destination_bucket=static_bucket,
    distribution=cdn,
) 'SlsBlogStaticS3Deployment' 'website_static' Stack 2: API/backend Our backend consists of an API Gateway (REST) connected to a . Microservices receives a lot of press, but you probably shouldn’t always break your applications in several functions. A Monolith is just fine - -, really. monolithic Lambda function and sometimes recommended This API & Lambda support a single endpoint (with GET and POST methods) with a queryString “action”, which takes three parameters: : populates the latest blog posts get-latest-articles : triggered when someone likes an article like-article : post a new blog article publish-article Here’s the power of the CDK model. We can create a REST API with 10 lines of code: rest_api_blog = aws_apigateway.LambdaRestApi(
    self, ,
    handler=lambda_blog,  # Previously declared Lambda 'sls-blog-rest-api-gateway' = . ( ),
    ),
) function deploy_options aws_apigateway StageOptions stage_name= ,
        throttling_rate_limit=lambda_param_max_concurrency,
        logging_level=aws_apigateway.MethodLoggingLevel( 'api' 'INFO' One nice little thing is that Lambda memory is used as a cache for the latest articles. We load the cache container outside the Lambda handler function. It remains in memory even after an invocation ends and is available for subsequent requests. Learn more here about . how to use Lambda as a cache mechanism MAX_CACHE_AGE: int = # In seconds
CACHE_LATEST_ARTICLES: Dict[str, Union[int, list]] = { : time.time(), : [],
} 120 'last_update' 'articles' All the data is stored in DynamoDB (DDB) using a , in . The site only shows the latest blog articles and items get after a few days by setting a attribute. single-table design on-demand mode auto-deleted by DDB time-to-live ddb_attr_time_to_live = ddb_param_max_parallel_streams = ddb_table_blog = aws_dynamodb.Table(
    self, ,
    partition_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.STRING,
    ),
    billing_mode=aws_dynamodb.BillingMode.PAY_PER_REQUEST,
    point_in_time_recovery=True,
    removal_policy=core.RemovalPolicy.DESTROY,
    time_to_live_attribute=self.ddb_attr_time_to_live,
    stream=aws_dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
) 'time-to-live' 5 'sls-blog-dynamo-table' 'id' The DDB table also has a that makes it easier to retrieve articles ordered by date for the site: GSI (Global Secondary Index) self.ddb_table_blog.add_global_secondary_index(
    index_name= ,
    partition_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.STRING,
    ),
    sort_key=aws_dynamodb.Attribute(
        name= ,
        type=aws_dynamodb.AttributeType.NUMBER,
    ),
    projection_type=aws_dynamodb.ProjectionType.ALL,
) 'latest-blogs' 'item-type' 'publish-timestamp' Modifications to DDB items generate streams that are processed by a . These streams are then repackaged and sent to a Kinesis Firehose stream processor. second Lambda function DDB doesn’t provide the flexibility that SQL databases offer and many choose , for example. Although Aurora is a great service, personally I prefer DDB for its simplicity and reliable, consistent performance. But sometimes we do need to run analytical queries, those with aggregations and on-the-fly filters. For that, we’ll be using Athena (more in the next Stack). Aurora Serverless Stack 3: analytical querying A Kinesis Firehose Stream is responsible for batching data inserted/modified in DDB, converting them to Apache Parquet format and storing in dedicated S3 buckets. From S3, we create a Data Lake with AWS Glue (used to declare our data schemas) and Athena (used to query the data). Athena is extremely powerful. We can statements ( ) to query terabytes of data and pay on-demand ($0.005 per GB of data scanned). Using Parquet not only improves query speed, but also reduces cost by minimizing the amount of data Athena needs to scan for each query. use SQL SELECT with some limitations Queries that are impossible or expensive/slow in DynamoDB, such as aggregations and JOINs, are fast and cheap in Athena. The two services combine each other in a perfect way, so that your application has optimized transactional storage and flexible analytical querying capabilities. We can use Athena to query all articles ever published and cross-reference with likes and HTTP metadata (source IP address, country, device type, etc). Even articles that were already expired by DynamoDB TTL (time-to-live) would continue to be available in the Data Lake. For example, which countries are liking the most articles? In the AWS Console, we get something like this: Queries can also be executed programmatically with or (e.g. ) to integrate anywhere we need this data. Athena API AWS SDKs Python’s boto3 Athena also supports JOINs. Here’s an example joining articles and HTTP metadata to analyze the most popular authors among readers of a particular country: Deployment CDK can deploy one Stack at a time. Since we have three, it’s necessary to specify which one when running the command. We do that by passing in the Stack ID as a CLI argument. For example, the following command will deploy the SlsBlogApiStack (id: ): CDK deploy sls-blog-api cdk deploy sls-blog-api Since all Stacks involve some type of permission granting, CDK asks for confirmation before deploying those resources. You can review the permissions requested and hit y when it’s good to go. Wrapping up We’ve covered how to structure CDK apps and add a bunch of AWS Resources to deploy with a simple command. CDK deploy If you’re new to the CDK - and as suggested early in the article -, it’s strongly recommended to follow AWS and . CDK workshop documentation Keep an eye on future publications as well, as is releasing other examples and tutorials to reap the most out of AWS serverless services with the power of infra automation with CDK or else. Dashbird Previously published at https://dashbird.io/blog/crash-course-aws-cdk-serverless-rest-api-data-lake-analytical-querying/

Quick Guide to AWS CDK and Serverless with REST API and Data Lake Analytical Querying

The exact difference between the Data Warehouse and AWS Data Lake. Let me demonstrate what AWS Data Lake is all about and depict the reason for it’s popularity. In the world of mazon eb ervice ( ), Amazon S3 is an amazing object container. Like any bucket, you can put content in it in a neat and orderly fashion, or you can just dump it in. But no matter how the data gets there, once it’s there, you need a way to organize it in a meaningful way so you can find it when you need it. This is where data lakes come in. A W S AWS AWS Data Lake A data lake is a centralized repository that allows you to store structured, semi-structured, and unstructured data at any scale. A data lake is an architectural concept that helps you , and , through a single set of tools. A data lake takes Amazon S3 buckets and inside the buckets. It doesn’t matter how the data got there or what kind it is, you can store both structured and unstructured data effectively in an Amazon S3 data lake AWS offers a set of tools to manage the entire Data Lake without treating each bucket as separate, unassociated objects manage multiple data types from multiple sources both structured unstructured organizes them by categorizing the data On-Premises Data Movement Data lakes allow you to . Data is collected from multiple sources and moved into the data lake in it’s original format. The process allows you to scale the data of , while from defining data structures, schemas and transformation. import any amount of data any size saving time Real-time Data Movement Data lakes allow you to import . Data can be collected from multiple stream data sources and moved into the data lakes in its original format. any amount of data that can come in real time Machine Learning Data lakes enables organizations to including reporting on and implementing where are built to forecast likely outcomes and suggest a range of prescribed actions to achieve the optimal result. generate different types of insights historical data machine learning models Analytics Data lakes allow various roles in the organization, such as Data Scientists, Data Developers and Business Analysts, to access data with their choice of analytic tools and frameworks. This includes such as Hadoop, Presto, and Apache Spark and commercial offerings from data warehouse and BI vendors. open source frameworks Data lakes allow you to run analytics system. without the need to move your data to a separate analytics Benefits of a data lake on AWS Are a solution. You can durably store a nearly unlimited amount of data using Amazon S3. cost-effective data storage Implement industry-leading . AWS uses stringent data security, compliance, privacy, and protection mechanisms. security and compliance Allow you to take advantage of to ingest data into your data lake. many different data collection and ingestion tools Help you to simply and efficiently. Use AWS Glue to understand the data within your data lake, prepare it, and load it reliably into data stores. Once AWS Glue catalogs your data, it is immediately searchable, can be queried, and is available for ETL processing. categorize and manage your data Help you turn data into . Harness the power of purpose-built analytic services for a wide range of use cases, such as interactive analysis, data processing using Apache Spark and Apache Hadoop, data warehousing, real-time analytics, operational analytics, dashboards, and visualizations. meaningful insights Business Problem Many businesses end up grouping data together into numerous storage locations called silos. These silos are rarely managed and maintained by the same team, which can be problematic. Inconsistencies in the way data was written, collected, aggregated, or filtered can cause problems when it is compared or combined for processing and analysis. For example, one team may use the address field to store both the street number and street name, while another team might use separate fields for street number and street name. When these datasets are combined, there is now an inconsistency in the way the address is stored, and it will make analysis very difficult. AWS Solution But by using Data Lakes, you can (a repository that contains raw data that is accessible by one department but isolated from the rest of that organization) and that is managed by a single team. That gives you a single, . Because data can be stored in its raw format, you don’t need to convert it, aggregate it, or filter it before you store it. Instead, you can leave that pre-processing to the system that processes it, rather than the system that stores it. break down data silos bring data into a single, central repository consistent source of truth In other words, you to make it usable. You , however it got there, however it was written. When you’re talking exabytes of data, you can’t afford to pre-process this data in every conceivable way it may need to be presented in a useful state. don’t have to transform the data keep the data in its original form Let’s talk about having a . When we talk about truth in relation to data, we mean the . Is it what it should be? Has it been altered? Can we validate the chain of custody? When creating a single source of truth, we’re creating a dataset, in this case . The bonus is that we know it to be and . It’s trustworthy. single source of truth trustworthiness of the data the data lake, which can be used for all processing and analytics consistent reliable Amazon S3 for a solution meeting these requirements and tools for analyzing the data without requiring movement. Data Lakes provide a single storage backbone Why Data Lake is Popular? As the volume of data has increased, so have the options for storing data. Traditional storage methods such as However, . These new options can confuse businesses that are trying to be and . data warehouses are still very popular and relevant. data lakes have become more popular recently financially wise technically relevant : data warehouses or data lakes? . They are different solutions that can be used together while . So which is better Neither and both to maintain existing data warehouses taking full advantage of the benefits of data lakes Data Warehouse A data warehouse is a coming from one or more data sources. Data flows into a data warehouse from transactional systems, relational databases, and other sources. These data sources can include structured, semistructured, and unstructured data. central repository of information . Data is within the data warehouse . A schema defines how data is stored within tables, columns, and rows. The to ensure integrity of the data. The transformation process often involves the steps required to make the source data conform to the schema. These data sources are transformed into structured data before they are stored in the data warehouse stored using a schema schema enforces constraints on the data Following the first successful of data into the data warehouse, the process of ingesting and the data can continue at a regular cadence. Business analysts, data scientists, and decision makers access the data through business intelligence (BI) tools, SQL clients, and other analytics applications. Businesses use , , and tools to extract insights from their data, , and . These reports, dashboards, and analytics tools are powered by data warehouses, which store data efficiently to minimize I/O and deliver query results at blazing speeds to hundreds and thousands of users concurrently. ingestion transforming reports dashboards analytics monitor business performance support decision making Comparison of Data Warehouse and Data Lake Analyzing a Data Warehouse For analysis to be most effective, it should be performed on data that has been processed and cleansed. This often means This data is then placed in a data warehouse. It is very common for data from many different parts of the organization to be combined into a single data warehouse. implementing an ETL operation to collect, cleanse, and transform the data. Analyzing a Data Lake Data lakes provide customers a means for including unstructured and semistructured data in their analytics. . This beyond the confines of a single data warehouse. Analytic queries can be run over cataloged data within a data lake extends the reach of analytics Businesses can , with high , , at and at . Businesses can easily access and analyze data in a variety of ways using the tools and frameworks of their choice in a high-performance, cost-effective way without having to move large amounts of data between storage and analytics systems. securely store data coming from applications and devices in its native format availability durability low cost, any scale I hope the above content is knowledgeable and would have given you a glance about the topic. Do follow me on & to get updates regarding all my blogs. If you really enjoy this post, then do show your love by banging the because learning has no limits. Medium LinkedIn Claps Button below Thank you for reading…!!

Why Is Amazon AWS Data Lake Gaining Popularity?

If you are following the trends in data science, it is more likely that you have heard the words big data, analytics, and machine learning. These days everyone wants to jump into this area of data science. Many of the software giants like Google, Amazon, Microsoft & etc. are already leading the way. However, it’s not that easy for a new business to enter this area of expertise due to many reasons. One of the core problems is that the data is scattered everywhere in different systems and their own databases. It’s likely these datasets will live for many years, hardly providing any value for its businesses. Why Not Create a Data Warehouse Instead? Although it would be wonderful if we can create a data warehouse in the first place (Check my article on for more details). Things to consider before building a serverless data warehouse However, there are several practical challenges in creating a data warehouse at a very early stage for business. One of the main reason is that it is difficult to know exactly which data sets are important and how they should be cleaned, enriched, and transformed to solve different business problems. Just imagine how much effort you need to put to identify data sets upfront from all the different systems and extract these data, doing the cleaning, enriching and transforming them such that they could be pushed into the data warehouse at a very early stage. Unless you have the data science experts in-house who also understand the business domain in depth to identify the required data sets and do the data preparations, it’s highly likely to screw up the things. What is a Data Lake? A data lake is a centralized repository to store all the structured and unstructured data. The real advantage is of a data lake is, it is possible to store data as-is where you can immediately start pushing data from different systems. These data could be in CSV files, Excel, Database queries, Log files & etc. could be stored in the data lake with the associated metadata without having to first structure the data. Once the data is available in the data lake over a time period, it’s possible to process the data later to run different types of analytics and big data processing for data visualization. It is also possible to use the data from the data lake for and for better-guided decisions. machine learning deep learning tools Creating a Data Lake for your Business For a business, to start creating a data lake and making sure that different data sets are added consistently over long periods of time requires a process and automation. To move in this direction, the first thing is to select a data lake technology and relevant tools to set up the data lake solution. 1. Setup a Data Lake Solution If you plan to create a data lake in a cloud, you can deploy a which uses serverless services underneath without incurring a huge cost upfront and a significant portion of the cost of data lake solution is variable and increases mainly based on the amount of data you put in. data lake on AWS 2. Identify Data Sources Then it is important to identify the data sources and the frequency of data being added to the data lake. Once the data sources are identified, make sure that the decisions are taken to either add the data sets as-is or to do the required level of cleaning and transformation of the data. It is also important to identify the metadata for individual types of data sets. 3. Establish Processes and Automation Since the data sets are is coming from different systems which might even be belonging to different departments of the business, it’s important to establish processes for consistency. For example, the HR department could be informed to publish the employee satisfaction after each survey which is carried out annually to the data lake. Another example is that the Account department publishes data on payroll monthly to the data lake. For operations that require a higher frequency of data publishing or time-consuming work, it is possible to automate the data sourcing process. This could involve, automating the extraction, transformation, and publishing of data to the data lake or at least automate some of the individual steps. 4. Ensure Right Governance After setting up the data lake, it’s important to make sure, that the data lake is functioning properly. It’s not only about putting data into the data lake but also to allow or to facilitate the data retrieval for other systems to generate data-driven informed business decisions. Otherwise, the data lake will end up as a data swamp in the long run with little to no use. 5. Using the Data from Data Lake After the data lake is properly set up and functioning for a reasonable period, you will be already collecting data to your data lake with the right amount of associated metadata. It will require to implement different processes with ETL (Extract Transform and Load) operations before using them to drive different business decisions. This is where the importance of Data Warehouses and Data Visualization tools come in. You can either publish the data to a Data Warehouse if there are more processing needs to be done in correlation with different data sets from other systems or directly feed into Data Visualization and Analytic tools like and . Microsoft Power BI AWS QuickSight So What’s Next? The most important thing comes next is to ask the right business questions which could be answered based on the availability of the data. Although it seems like it is too obvious, this is one of the areas many businesses make things so complex. Although there is a fully functioning data lake which produces useful insights for the business, it is important not to stop from there. The strength of a data lake lies with continuous development and the evaluation of the solutions.

What is a Data Lake and How to Create One for Your Business

There is a bit of a confusion between Data Warehouse vs Data Lake or ETL vs ELT. I hear that Data Warehouses are not used anymore, that they are replaced by Data Lakes altogether, but is that true? And why do we need Data Warehouses anyway? I will go into that as well as the definitions of both pluses explain the differences between them. Data Warehouse vs Data Lake Data Warehouse definition A Data Warehouse, in short DWH and also known as an Enterprise Data Warehouse (EDW), is the traditional way of collecting data as we do . The DWH serves the purpose of being the data integration from many different sources, the single point of truth and the data management meaning cleaning, historize and data joined together. It provides greater executive insight into corporate performance with management Dashboards, Reports or Ad-Hoc Analyses. since 31 years Various types of business data are analysed with Data Warehouses. The need for it often becomes evident when analytic requirements run afoul of the ongoing performance of operational databases. Running a complex query on a database requires the database to enter a temporarily fixed state. This is often untenable for transactional databases. A data warehouse is employed to do the analytic work, leaving the transactional database free to focus on transactions. The other characteristics are the ability to analyse data from multiple origins (e.g. your Google Analytics with your CRM data), and that is highly transformed and structured due to the ETL (Extract Transform Load) process. Data Lake definition A Data Lake is a store full of unstructured and structured data, stored as-is, without a specific purpose in mind, that can be built on multiple technologies such as Hadoop, NoSQL, Amazon Simple Storage Service, a relational database, or various combinations and different formats (e.g. Excel, CSV, Text, Logs, etc.). According to , the data lake arose because new types of data needed to be captured and exploited by the enterprise. As this data became increasingly available, early adopters discovered that they could extract insight through new applications built to serve the business. The data lake supports the following capabilities: Hortonworks Data Lake Whitepaper To capture and store raw data at scale for a low cost To store many types of data in the same repository To perform transformations on the data To define the structure of the data at the time, it is used, referred to as schema on reading To perform new types of data processing To perform single subject analytics based on particular use cases Differences between Data Warehouse and Data Lake I like the definition and comparison by James Dixon, founder and CTO of Pentaho: If you think of a DWH as a store of bottled water — cleansed and packaged and structured for easy consumption — the data lake is a large body of water in a more natural state . The contents of the data lake stream in from a source to fill the lake and various users of the lake can come to examine, dive in, or take samples. Different analogies As you can see on the picture below, both Technologies are created for different purposes: When to use what? As mentioned in the introduction, companies are , although it’s two different things, it still can make sense. Especially when you want real-time data, as the Data Warehouse typically works in batch processes, the Data Lake works near real time and handling Big Data. It’s made for huge data and stores them unstructured easy and fast. So when should I use what? shifting from the Data Warehouse to the Data Lake is designed for : The Data Warehouse slowly changing data + daily summaries, weekly summaries and monthly summaries of known + structured data + easy and fast access to many operational business users on the other side is designed for The Data Lakes quickly changing data + data that tells you what happened one minute or five minutes ago + raw, un- and semi-structured data + easy and fast access to a few superpower users and Data Scientists Modern Data and Analytics Environment It is common, especially in mid or large size organisation to have both environments. The image below illustrates how you would integrate it with an Enterprise Data Warehouse and a Data Lake: © Dell EMC ETL vs ELT ETL (Extract Transform and Load) and ELT (Extract Load and Transform) is what has described above. . ETL is what happens within a Data Warehouse and ELT within a Data Lake ETL is the most common method used when transferring data from a source system to a . In that process, you load data to your stage-layer of your DWH, and transform it to the (Facts and Dimensions) and at the end, you load it to a final Data Mart or a Cube for further Data Visualisations. Data Warehouse clean Dimensional Model If you want to use ELT, that’s when you want to build a . You extract data, mostly done by physical files, load it into your Data Lake in your Cloud Storage and only then start transforming and cleaning the data. The natural process is that you begin exploring and analysing your data and find out, that the data is dirty and then you begin cleaning it. Data Lake Big Data How is connected to this topic? Big Data is more or less gathering massive amounts of data (several million rows per second) from devices like IoT (Internet of Things), different data points from each smartphone, etc. With specific Big Data infrastructure and algorithms (e.g. map-reduce) you collect the data and store it into the Data Lake. Big data is greatest used with a Data Lakes. Big Data As a new technology arises an old one gets replaced, that’s maybe why They say it’s no longer relevant . But why? The question is, are you still have a traditional Data Warehouse with an ETL tool, are you 100% happy with it? There is a great chance that you are not. Because of a significant amount of coding required in traditional ETL tools, your ETL tool was probably outdated before you were ready to deploy it. many technologists and thought leaders are declaring the Data Warehouse is dead. in the age of Big Data But these prognosticators are mistaken. . It is not a Data Warehouses that are dead, but the traditional way of designing and building them. A better and new approach is using Data Warehouse Automation (DWA) Tools that automates the recurring parts of developing a Data Warehouse to . Find more information in the or directly . Big data can extend and enrich a Data Warehouse, but cannot replace it cut down time development time by 40–60% DWA Blog post series why we should use DWA tools Data Vault Another way of adapting to Big Data and fast-changing data connection points is the modelling and methodology which enables you a more dynamic and flexible way to implement additions to your Data Warehouse. Lately, , because it addresses the elements of the problems we identified within Data Warehousing: Data Vault there has been an interesting move to use a Data Vault as a governed Data Lake It adapts to a changing business environment It supports huge data sets It simplifies the Data Warehouse design complexities It increases usability by business users because it models after the business domain It allows for new data sources to add without impacting the existing design This technological advancement is already proving to be highly effective and efficient. Easy to design, build, populate, and change. New Data Warehouses should be created with this methodology, also a point where helps you. Data Warehouse Automation Conclusion — Will a Data Lake replace the Data Warehouse? I don’t think so. As elaborated thoroughly above, it will still need both over a long period. Both are designed for distinctive purposes and have different advantages. And I believe we should be careful with Data Lakes as more data is always better but can also lead to more chaotic data stores where nobody knows the what’s in there and it takes everyone a lot of time just to get the needed data. In my opinion, it makes sense to have both environments to be flexible for fast changes and analysis within the Data Lake. But still be able to make very well structured Analytics with Dashboards and Reports for the operational business user. That’s why I would suggest importing patterns and essential analysis you found in the Data Lake back to your Data Warehouse to make it easily accessible for everyone in a standard way and tools users know. Thanks for reading that far. Please feel free to add comments or anything you don’t agree. Originally published at www.sspaeti.com on October 26, 2017.

Data Warehouse vs Data Lake

What is Data Collection and Ingestion? Data Collection and Data Ingestion are the processes of fetching data from any data source which we can perform in two ways - Real-time Streaming Batch Streaming In Today’s World, Enterprises are generating data from different Sources and building Real Time Data lake; we need to Integrate various sources of Data into One Stream. In this Blog We are sharing how to Ingest, Store and Process Data using Apache Nifi and in Coming Blogs, we will be Sharing Data Collection and Ingestion from Below Sources Twitter Data ingestion From Logs Data Ingestion from IoT Devices Data Collection and Ingestion from RDBMS (e.g., MySQL) Data Collection and Ingestion from ZiP Files Data Collection and Ingestion from Text/CSV Files Objectives for the Data Lake A Central Repository for Management Big Data Reduce costs by offloading analytical systems and archiving cold data Testing Setup for experimenting with new technologies and data Automation of Data pipelines MetaData Management and Catalog Tracking measurements with alerts on failure or violations Data Governance with clear distinction of roles and responsibilities Data Discovery, Prototyping, and experimentation Introduction to Apache NiFi provides an easy to use, the powerful, and reliable system to process and distribute the data over several resources. Apache NiFi Apache NiFi is used for routing and processing data from any source to any destination. The process can also do some data transformation. It is a UI based platform where we need to define our source from where we want to collect data, processors for the conversion of the data, a destination where we want to store the data. Each processor in NiFi have some relationships like success, retry, failed, invalid data, etc. which we can use while connecting one processor to another. These links help in transferring the data to any storage or processor even after the failure by the processor. Benefits of Apache NiFi Real-time/Batch Streaming Support both Standalone and Cluster mode Extremely Scalable, extensible platform Visual Command and Control Better Error handling Core Features of Apache NiFi — A core philosophy of NiFi has been that even at very high scale, guaranteed delivery is a must. It is achievable through efficient use of a purpose-built persistent write-ahead log and content repository. Guaranteed Delivery — NiFi supports buffering of all queued data as well as the ability to provide back pressure as those lines reach specified limits or to an age of data as it reaches a specified age (its value has perished). Data Buffering / Back Pressure AND Pressure Release NiFi allows the setting of one or more prioritization schemes for how data from a queue is retrieved. The default is oldest first, but it can be configured to pull newest first, largest first, or some other custom scheme. Prioritized Queuing — There are points of a data flow where the data is critical, and it is less intolerant. There are also times when it must be processed and delivered within seconds to be of any value. NiFi enables the fine-grained flow particular configuration of these concerns. Flow Specific QoS — NiFi automatically records, indexes, and makes available provenance data as objects flow through the system even across fan-in, fan-out, transformations, and more. This information becomes extremely critical in supporting compliance, troubleshooting, optimization, and other scenarios. Data Provenance — NiFi’s content repository is designed to act as a rolling buffer of history. As Data ages off, it is removed from the content repository or as space is needed. Recovery / Recording a rolling buffer of fine-grained history — NiFi enables the visual establishment of data flows in real-time. And provide UI based approach to build different data flow. Visual Command and Control — It also allows us to create templates of frequently used data streams. It can also help in migrating the data flows from one machine to another. Flow Templates — — NiFi supports Multi-tenant Authorization. The authority level of a given data flow applies to each component, allowing the admin user to have a fine grained level of access control. It means each NiFi cluster is capable of handling the requirements of one or more organizations. Security With NiFi we can move data to multiple destinations at one time. After processing the data stream, we can route the flow to the various destinations using NiFi’s processor. It can be helpful when we need to back our data on multiple destinations. Parallel Stream to Multiple Destination — NiFi Clustering When we require moving a large amount of data, then the only single instance of NiFi is not enough to handle that amount of data. So to handle this we can do clustering of the NiFi Servers, this will help us in scaling. We just need to create the data flow on one node, and this will make a copy of this data flow on each node in the cluster. NiFi introduces Zero-Master Clustering paradigm in Apache NiFi 1.0.0. A previous version of Apache NiFi based upon a single “Master Node” (more formally known as the NiFi Cluster Manager). If the master node gets lost, data continued to flow, but the application was unable to show the topology of the flow, or show any stats. But in Zero-Master we can make changes from any node of the cluster. And if master node disconnects, then automatically any active node is elected as Master Node. Each node has the same the data flow, so they work on the same task as the other nodes are working, but each operates on the different datasets. In NiFi cluster, one node is elected as the Master(Cluster Coordinator), and another node sends heartbeats/status information to the master node. This node is responsible for the disconnection of the other nodes that do not send any pulse/status information. This election of the master node is done via . And In the case when the master nodes get disconnected, Apache Zookeeper elects any active node as the master node. Apache Zookeeper Data Collection and Ingestion from Twitter using Apache NiFi to Build Data Lake Amazon S3 Amazon Redshift Apache Kafka Apache Spark Apache Flink ElasticSearch Apache Hive Fetching Tweets with NiFi’s Processor NiFi’s ‘ ’ processor is used to fetch tweets. It uses Twitter Streaming API for retrieving tweets. In this processor, we need to define the endpoint which we need to use. We can also apply filters by location, hashtags, particular IDs. GetTwitter Here we can set the endpoint from which data should get pulled. Available parameters - Twitter Endpoint — — Fetch public tweets from all over the world. - Sample Endpoint — This is same as streaming API, but it ensures 100% guarantee delivery of tweets with filters. - Firehose Endpoint — If we want to filter by any hashtags or keywords - Filter Endpoint Consumer key provided by Twitter. Consumer Key — Consumer Secret provided by Twitter. Consumer Secret — Access Token provided by Twitter. Access Token — Access Token Secret provided by Twitter. Access Token Secret — Languages for which tweets should fetch out. Languages — Hashtags for which tweets should fetch out. Terms to Filter — Twitter user IDs that should be followed. IDs to follow — Now processor is ready for transmission of the data(tweets). From here we can move our data stream to anywhere like Amazon S3, Apache Kafka, ElasticSearch, Amazon Redshift, , Hive, , etc. NiFi can move data multiple destinations parallelly. GetTwitter HDFS Cassandra Data Integration Using Apache NiFi and Apache Kafka For this, we are using NiFi processor . ‘PublishKafka_0_10’ In the Scheduling Tab, we can configure how many concurrent tasks to be executed and schedule the processor. In Properties Tab, we can set up our Kafka broker URLs, topic name, request size, etc. It will write data to the given topic. For the best results, we can create a Kafka topic manually of a defined partitions. Apache Kafka can be used to process data with Apache Beam, Apache Flink, Apache Spark. Data Integration Using Apache NiFi to Amazon RedShift with Amazon Kinesis Firehose Stream Now we integrate Apache NiFi to Amazon Redshift. NiFi uses Amazon Kinesis Firehose Delivery Stream to store data to Amazon Redshift. This delivery Stream should get utilized for moving data to Amazon Redshift, Amazon S3, Amazon ElasticSearch Service. We need to specify this while creating Amazon Kinesis Firehose Delivery Stream. Now we have to move data to Amazon Redshift, so firstly we need to configure Amazon Kinesis Firehose Delivery Stream. While delivering data to Amazon Redshift, firstly the data is provided to Amazon S3 bucket, and then Amazon Redshift Copy command is used to move data to Amazon Redshift Cluster. We can also enable data transformation while creating Kinesis Firehose Delivery Stream. In this, we can also backup the data to another Amazon S3 bucket other than an intermediate bucket. So for this, we will use processor This processor will use that Kinesis Firehose stream for delivering data to Amazon Redshift. Here we will configure AWS credentials and Kinesis Firehose Delivery Stream. PutKinesisFirehose. Data Integration Using Apache NiFi to Amazon S3 sends data to both Amazon Redshift and uses Amazon S3 as the intermediator. Now if someone only wants to use Amazon S3 as the storage so NiFi can also use for sending data to Amazon S3 only. PutKinesisFirehose For this, we need to use NiFi processor In it, we have to configure our AWS credentials, bucket name, and path, etc. PutS3Object. Partitioning in Amazon S3 Bucket Most important aspect while storing data in S3 is partitioning. Here we can partition our data using expression language in the object key field. So Right now we have used day wise partitioning. Continue Reading The Full Article At — XenonStack.com/Blog

Data Ingestion Using Apache Nifi For Building Data Lake Using Twitter Data

Building Serverless Data Lake with AWS Glue DynamoDB and Athena

Using Arrow Flight SQL Protocol in Apache Doris 2.1 For Super Fast Data Transfer

We are back with another article for the HackerNoon Company of the Week series! Every week we feature an awesome tech brand from our tech company database, making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike.This week, we present our business blogging partnerMinIO - a high-performance, S3 compatible object store, built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO also offers features like data replication, lifecycle management and high availability, making it a top choice for modern cloud-native applications. Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon! Meet MinIO: #FunFact Do you know that MinIO is open source under GNU AGPL v3? Open source is the key driver of technological innovation and here's how the company shares its support and belief in the role of open source in AI revolution:"Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." MinIO’s Top Stories on HackerNoon MinIO has published 43 stories via HackerNoon Business Blogging Program, and has generated 2 years 23 days and 29 minutes of reading time! Here are our top 3 favorite stories by MinIO: Open-Source: The Next Step in AI Revolution
Quantum State: How Two Things Can Be True at the Same Time
Analyzing the Pros, Cons, and Risks of LLMs We are back with another article for the HackerNoon Company of the Week series ! Every week we feature an awesome tech brand from our tech company database , making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. This week, we present our business blogging partner MinIO - a high-performance, S3 compatible object store, built for large scale AI/ML, data lake and database workloads. It runs on-prem and on any cloud (public or private) and from the data center to the edge. MinIO also offers features like data replication, lifecycle management and high availability, making it a top choice for modern cloud-native applications. Company of the Week series Company of the Week tech company database tech company database MinIO MinIO Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon! Wanna be part of HackerNoon’s evergreen tech company database? Request Your Tech Company Page on HackerNoon ! Request Your Tech Company Page on HackerNoon Request Your Tech Company Page on HackerNoon Meet MinIO: #FunFact Do you know that MinIO is open source under GNU AGPL v3? Open source is the key driver of technological innovation and here's how the company shares its support and belief in the role of open source in AI revolution : "Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." role of open source in AI revolution role of open source in AI revolution "Embracing a philosophy of extreme open innovation, companies that truly participate in open-source software development challenge traditional notions of competitive advantage by acknowledging that not all good code or great ideas reside within their organization. This shift supports the argument that shared innovations within the open-source ecosystem lead to faster market growth, providing even smaller software firms with more limited R&D funds the opportunity to benefit from R&D spillovers present in open-source software. This is because, in contrast to traditional outsourcing, open innovation enhances internal resources by leveraging the collective intelligence of the community, without diminishing internal R&D efforts. Meaning that open-source software companies don’t have to sacrifice their budgets to pursue thought leadership and code outside their organization." MinIO’s Top Stories on HackerNoon MinIO has published 43 stories via HackerNoon Business Blogging Program , and has generated 2 years 23 days and 29 minutes of reading time! Here are our top 3 favorite stories by MinIO: MinIO MinIO HackerNoon Business Blogging Program HackerNoon Business Blogging Program Open-Source: The Next Step in AI Revolution Quantum State: How Two Things Can Be True at the Same Time Analyzing the Pros, Cons, and Risks of LLMs Open-Source: The Next Step in AI Revolution Open-Source: The Next Step in AI Revolution Quantum State: How Two Things Can Be True at the Same Time Quantum State: How Two Things Can Be True at the Same Time Analyzing the Pros, Cons, and Risks of LLMs Analyzing the Pros, Cons, and Risks of LLMs

Meet MinIO: The HackerNoon Company of the Week

মিট মিনিও: সপ্তাহের হ্যাকারনুন কোম্পানি

Lernen Sie MinIO kennen: Das HackerNoon-Unternehmen der Woche

Conozca MinIO: la empresa de la semana de HackerNoon

Rencontrez MinIO : l'entreprise HackerNoon de la semaine

मिलिए मिनियो से: इस सप्ताह की हैकरनून कंपनी

MinIO のご紹介: 今週の HackerNoon 企業

MinIO를 만나보세요: 금주의 HackerNoon 회사

Conheça MinIO: a empresa HackerNoon da semana

Встречайте MinIO: компания недели HackerNoon

MinIO ile Tanışın: Haftanın HackerNoon Şirketi

Gặp gỡ MinIO: Công ty HackerNoon của tuần

认识 MinIO：本周 HackerNoon 公司

In this article, I let Madiha tell her story. With years of experience as a senior data engineer in various companies, her expertise goes beyond lines of code. She helped small and mid-sized companies build their reliable, scalable, cost-effective data platform. As we talk, Madiha tells her story, shares one of her most challenging projects, and gives advice on climbing the career ladder. Madiha's Background Story Madiha works as a Lead Data Engineer and certified Data Platform mentor for startups and aspiring engineers. She inspires startups to recognize the potential of data platforms in the age of AI in their early stage — not just from a technical standpoint, but also from the perspective of building cost and performance-optimized and trustworthy solutions for their data warehouse or data lake. She generously donates her time to mentor aspiring data engineers in addition to her regular duties. Her dedication to helping others succeed sets her apart as a true leader in the field. My current goal is two-fold: first, to raise awareness in the business about the need for a Data Platform at the Early Stage and second, to mentor the new generation of Data Engineers." — Madiha said. Madiha Khalid's foray into technology began in high school in Lahore, Pakistan. Faced with the choice between biology and computer science, she decided to dive into the less-trodden path of computer science. About 95% of our students chose to go to biology classes and the rest — to computer sciences. It was difficult for me back then, but I made my choice", — Madiha said. Despite initial setbacks and the challenge of being denied admission to chemical engineering, she shifted to computer science in early college; Madiha's passion for programming ignited during her high school years. Then, her journey continued into college, following in her sister's footsteps and pursuing a bachelor's in computer science. "I remember my first achievement in 2011 when I developed Image Compression Software for my Image Processing course Assignment. This application reduced the size of .jpg without compromising the real pixel quality, and it became quite popular among my University Friends. This is how I began my career journey towards becoming a  Software Engineer" — Madiha said Her early exposure to website development and freelancing projects marked the beginning of her hands-on experience. Madiha's dedication and late-night coding sessions secured her academic success with a 3rd position in BS and the Erasmus Mundus Scholarship for a competitive Master's in IT4BI, laying the foundation for her future career in the Data Engineering industry. Advice to Boost Personal Growth for the Aspiring Engineers For those aspiring to enter the tech field, Madiha advocates building skills with real-world business use cases. "Employers mostly look for seniors profiles who can solve their current challenges. They want to hear your stories about your impact and the contribution you made to the latest experiences, even though you are applying for Junior Position.
For a young professional, it's a good way to gain experience on your self-learning and apply these skills on some freelance or open-source projects, — Madiha said, — You can easily connect with the real business over Upwork or Fiverr. Contribution to open-source projects in your early career boosts your profile 10x to get your first dream job". Building a portfolio with real-world projects enhances credibility and storytelling during job interviews. There is no shortcut to learning skills in one night. It involves hands-on work and applying your learning skills to the real-world business use case.— Madiha said Maintaining an active LinkedIn profile and networking with industry professionals are crucial for visibility. Madiha's journey emphasizes the significance of perseverance, especially when faced with language barriers and cultural differences during international transitions. As the tech industry evolves rapidly, staying updated on new tools and integrations is essential. Finding one's niche within the extensive tech landscape is critical to a fulfilling career. Motivation Through Challenges Madiha's career trajectory took an exciting turn when she saw an internship at a major telecom company in Pakistan through a Facebook group. I think I had a lot of fortunate incidents during my career path. And this was one of them — fully accidental and without much belief in myself back then. — Madiha said. Her role involved automating processes for telecom engineers, setting the stage for her future in data engineering. From developing dashboards to optimizing data integration performance, Madiha's expertise grew. Her journey continued through a Master's in Business Intelligence in Europe, where she developed Big Data projects and deepened her knowledge as a Solution Architect. Studying at universities as an Erasmus student in Belgium and France and a pivotal internship at the Barcelona Supercomputing Center in Spain broadened her skills. Madiha's move to Germany marked the start of her role as a Data Engineer at Jimdo, where she eventually earned the position of Senior Data Engineer. Madiha's challenges in adapting to different cultures and managing complex data pipelines became stepping stones to her success. Her passion lies in building and optimizing data integration pipelines, a role she found immensely satisfying. Notable projects are the Data Migration of ~8 Billion Event Stream records using novel partitioning techniques from legacy systems to the Data Lake. Implement customer engagement metrics in near real-time with Apache Spark and make them directly available on customer dashboards. Creating the DWH GDPR Compliant with the data governance access layer. The list could continue, showcasing Madiha's impactful contributions to the field. Data Engineering Book Recommendations Here's a list of books and resources Madiha advises for anyone interested in big data: Beginners: Fundamentals of Data Engineering Building the foundation and concepts of Data engineering is very important, and this book's approach is the high level to build your base. Intermediate: The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling This book is a part of my Master's program and teaches data modeling, a highly demanded skill in job interviews. Intermediate:  Spark: The Definitive Guide [Book] This book provides the internal workings of Spark and hands-on sessions. It is written by the original creator of Apache Spark; hence, it is worth reading. Advance: Designing Data-Intensive Applications This was the first book Madiha read in 2017 while building her first real-time streaming application. It is highly recommended to understand how large-scale data applications develop. Insights for Aspiring Professionals In concluding her journey, Madiha Khalid shares valuable advice for young professionals entering the tech industry. Emphasizing the need for senior-level skills in the market, she recommends taking on short projects on freelance platforms and open-source projects to build a robust portfolio. Establishing a solid online presence through platforms like LinkedIn and GitHub aids networking and showcasing one's capabilities. Madiha's story is a testament to the power of determination, adaptability, and continuous learning in navigating the dynamic tech industry landscape. Aspiring professionals can draw inspiration from her journey and understand that challenges are not roadblocks but growth opportunities. In this article, I let Madiha tell her story. With years of experience as a senior data engineer in various companies, her expertise goes beyond lines of code. She helped small and mid-sized companies build their reliable, scalable, cost-effective data platform. As we talk, Madiha tells her story, shares one of her most challenging projects, and gives advice on climbing the career ladder. Madiha's Background Story Madiha works as a Lead Data Engineer and certified Data Platform mentor for startups and aspiring engineers. She inspires startups to recognize the potential of data platforms in the age of AI in their early stage — not just from a technical standpoint, but also from the perspective of building cost and performance-optimized and trustworthy solutions for their data warehouse or data lake. She generously donates her time to mentor aspiring data engineers in addition to her regular duties. Her dedication to helping others succeed sets her apart as a true leader in the field. My current goal is two-fold: first, to raise awareness in the business about the need for a Data Platform at the Early Stage and second, to mentor the new generation of Data Engineers." — Madiha said. My current goal is two-fold: first, to raise awareness in the business about the need for a Data Platform at the Early Stage and second, to mentor the new generation of Data Engineers." — Madiha said. My current goal is two-fold: first, to raise awareness in the business about the need for a Data Platform at the Early Stage and second, to mentor the new generation of Data Engineers." — Madiha said. Madiha Khalid's foray into technology began in high school in Lahore, Pakistan. Faced with the choice between biology and computer science, she decided to dive into the less-trodden path of computer science. About 95% of our students chose to go to biology classes and the rest — to computer sciences. It was difficult for me back then, but I made my choice", — Madiha said. About 95% of our students chose to go to biology classes and the rest — to computer sciences. It was difficult for me back then, but I made my choice", — Madiha said. Despite initial setbacks and the challenge of being denied admission to chemical engineering, she shifted to computer science in early college; Madiha's passion for programming ignited during her high school years. Then, her journey continued into college, following in her sister's footsteps and pursuing a bachelor's in computer science. "I remember my first achievement in 2011 when I developed Image Compression Software for my Image Processing course Assignment. This application reduced the size of .jpg without compromising the real pixel quality, and it became quite popular among my University Friends. This is how I began my career journey towards becoming a  Software Engineer" — Madiha said "I remember my first achievement in 2011 when I developed Image Compression Software for my Image Processing course Assignment. This application reduced the size of .jpg without compromising the real pixel quality, and it became quite popular among my University Friends. This is how I began my career journey towards becoming a  Software Engineer " — Madiha said "I remember my first achievement in 2011 when I developed Image Compression Software for my Image Processing course Assignment. This application reduced the size of .jpg without compromising the real pixel quality, and it became quite popular among my University Friends. This is how I began my career journey towards becoming a  Software Engineer — Madiha said Her early exposure to website development and freelancing projects marked the beginning of her hands-on experience. Madiha's dedication and late-night coding sessions secured her academic success with a 3rd position in BS and the Erasmus Mundus Scholarship for a competitive Master's in IT4BI, laying the foundation for her future career in the Data Engineering industry. Advice to Boost Personal Growth for the Aspiring Engineers For those aspiring to enter the tech field, Madiha advocates building skills with real-world business use cases. "Employers mostly look for seniors profiles who can solve their current challenges. They want to hear your stories about your impact and the contribution you made to the latest experiences, even though you are applying for Junior Position. For a young professional, it's a good way to gain experience on your self-learning and apply these skills on some freelance or open-source projects, — Madiha said, — You can easily connect with the real business over Upwork or Fiverr. Contribution to open-source projects in your early career boosts your profile 10x to get your first dream job". "Employers mostly look for seniors profiles who can solve their current challenges. They want to hear your stories about your impact and the contribution you made to the latest experiences, even though you are applying for Junior Position. "Employers mostly look for seniors profiles who can solve their current challenges. They want to hear your stories about your impact and the contribution you made to the latest experiences, even though you are applying for Junior Position. For a young professional, it's a good way to gain experience on your self-learning and apply these skills on some freelance or open-source projects, — Madiha said, — You can easily connect with the real business over Upwork or Fiverr. Contribution to open-source projects in your early career boosts your profile 10x to get your first dream job". For a young professional, it's a good way to gain experience on your self-learning and apply these skills on some freelance or open-source projects, — Madiha said, — You can easily connect with the real business over Upwork or Fiverr . Contribution to open-source projects in your early career boosts your profile 10x to get your first dream job". Upwork Upwork Fiverr Fiverr Building a portfolio with real-world projects enhances credibility and storytelling during job interviews. There is no shortcut to learning skills in one night. It involves hands-on work and applying your learning skills to the real-world business use case.— Madiha said There is no shortcut to learning skills in one night. It involves hands-on work and applying your learning skills to the real-world business use case . — Madiha said There is no shortcut to learning skills in one night. It involves hands-on work and applying your learning skills to the real-world business use case — Madiha said Maintaining an active LinkedIn profile and networking with industry professionals are crucial for visibility. Madiha's journey emphasizes the significance of perseverance, especially when faced with language barriers and cultural differences during international transitions. As the tech industry evolves rapidly, staying updated on new tools and integrations is essential. Finding one's niche within the extensive tech landscape is critical to a fulfilling career. Motivation Through Challenges Madiha's career trajectory took an exciting turn when she saw an internship at a major telecom company in Pakistan through a Facebook group. I think I had a lot of fortunate incidents during my career path. And this was one of them — fully accidental and without much belief in myself back then. — Madiha said. I think I had a lot of fortunate incidents during my career path. And this was one of them — fully accidental and without much belief in myself back then. — Madiha said. I think I had a lot of fortunate incidents during my career path. And this was one of them — fully accidental and without much belief in myself back then. — Madiha said. Her role involved automating processes for telecom engineers, setting the stage for her future in data engineering. From developing dashboards to optimizing data integration performance, Madiha's expertise grew. Her journey continued through a Master's in Business Intelligence in Europe, where she developed Big Data projects and deepened her knowledge as a Solution Architect. Studying at universities as an Erasmus student in Belgium and France and a pivotal internship at the Barcelona Supercomputing Center in Spain broadened her skills. Madiha's move to Germany marked the start of her role as a Data Engineer at Jimdo, where she eventually earned the position of Senior Data Engineer. Madiha's challenges in adapting to different cultures and managing complex data pipelines became stepping stones to her success. Her passion lies in building and optimizing data integration pipelines, a role she found immensely satisfying. Notable projects are the Data Migration of ~8 Billion Event Stream records using novel partitioning techniques from legacy systems to the Data Lake. Implement customer engagement metrics in near real-time with Apache Spark and make them directly available on customer dashboards. Creating the DWH GDPR Compliant with the data governance access layer. The list could continue, showcasing Madiha's impactful contributions to the field. Data Engineering Book Recommendations Here's a list of books and resources Madiha advises for anyone interested in big data: Beginners: Fundamentals of Data Engineering Beginners : Fundamentals of Data Engineering Beginners Fundamentals of Data Engineering Fundamentals of Data Engineering Building the foundation and concepts of Data engineering is very important, and this book's approach is the high level to build your base. Intermediate: The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling Intermediate : The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling Intermediate The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling This book is a part of my Master's program and teaches data modeling, a highly demanded skill in job interviews. Intermediate:  Spark: The Definitive Guide [Book] Intermediate : Spark: The Definitive Guide [Book] Intermediate Spark: The Definitive Guide [Book] Spark: The Definitive Guide [Book] This book provides the internal workings of Spark and hands-on sessions. It is written by the original creator of Apache Spark; hence, it is worth reading. Advance: Designing Data-Intensive Applications Advance : Designing Data-Intensive Applications Advance Designing Data-Intensive Applications Designing Data-Intensive Applications This was the first book Madiha read in 2017 while building her first real-time streaming application. It is highly recommended to understand how large-scale data applications develop. Insights for Aspiring Professionals In concluding her journey, Madiha Khalid shares valuable advice for young professionals entering the tech industry. Emphasizing the need for senior-level skills in the market, she recommends taking on short projects on freelance platforms and open-source projects to build a robust portfolio. Establishing a solid online presence through platforms like LinkedIn and GitHub aids networking and showcasing one's capabilities. Madiha's story is a testament to the power of determination, adaptability, and continuous learning in navigating the dynamic tech industry landscape. Aspiring professionals can draw inspiration from her journey and understand that challenges are not roadblocks but growth opportunities.

Unveiling the Code: Meet the Writer and Lead Data Engineer - Madiha Khalid

Getting Started With Apache Iceberg and Resources if You Would Like To Go Further

The modern data landscape demands a new breed of infrastructure – one that seamlessly integrates structured and unstructured data, scales effortlessly, and empowers efficient AI/ML workloads. This is where come in, providing a central hub for all your data needs. However, building and managing an effective data lake can be complex. modern datalakes This blog post dives deep into three powerful tools that can optimize your current approach: , and MinIO. The steps below will walk you through how these services seamlessly combine to create a robust, cloud-native data lake architecture specifically optimized for AI/ML workloads. Apache Iceberg Tabular, What is Tabular? Tabular is a data platform created by the original creators of Apache Iceberg. It is designed to provide an independent, universal storage platform that connects to any compute layer, eliminating data vendor lock-in. This feature is critical to the modern data stack, it allows users to pick best-in-class compute and storage tools without being forced into a particular vendor's aged and or mismatched tool set. In an of MinIO and Iceberg and can be enhanced by Tabular. Tabular can be used to manage and query Iceberg data stored in MinIO, allowing for the storage and management of structured data in a scalable, high-performance, and cloud-native manner. These Kubernetes native, components work together smoothly with very little friction and build on each other’s capabilities to perform at scale. architecture Why S3FileIO instead of Hadoop's file-io? This implementation leverages Iceberg's S3FileIO. S3FileIO is considered better than Hadoop's file-io for several reasons. Some of which we’ve already discussed : elsewhere : Iceberg's S3FileIO is designed to work with cloud-native storage. Optimized for Cloud Storage Iceberg uses an to distribute files across multiple prefixes in a MinIO bucket, which helps to minimize throttling and maximize throughput for S3-related IO operations. Improved Throughput and Minimized Throttling: ObjectStoreLocationProvider Iceberg has been to fully leverage strict consistency by eliminating redundant consistency checks that could impact performance. Strict Consistency: updated Iceberg's S3FileIO implements a algorithm, which uploads data files parts in parallel as soon as each part is ready, reducing local disk usage and increasing upload speed. Progressive Multipart Upload: progressive multipart upload Iceberg allows for for S3 API writes to ensure the integrity of uploaded objects, which can be enabled by setting the appropriate catalog property. Checksum Verification: checksum validations Iceberg supports adding to objects during write and delete operations with the S3 API, which can be useful for cost tracking and management. Custom Tags: custom tags The FileIO interface in Iceberg does not require as strict guarantees as a Hadoop-compatible FileSystem, which allows it to that could otherwise degrade performance. Avoidance of Negative Caching: avoid negative caching In contrast, Hadoop's S3A FileSystem, which was used before S3FileIO, does not offer the same level of optimization for cloud storage. All this to say: Don’t hobble your future-facing data lake infrastructure with the trappings of the past. Prerequisites Before you begin, make sure your system meets the following requirements: Docker Docker Compose If you're starting from scratch, you can install both using the installer for your specific platform. It is often easier than downloading Docker and Docker Compose separately. Verify if Docker is installed by running the following command: Docker Desktop docker-compose --version Getting started To get started, clone or copy the YAML file in Tabular’s . You just need the YAML for this tutorial. Feel free to explore the rest of the repository at a later time. git repository Breaking it Down The provided YAML file is a Docker Compose configuration file. It defines a set of services and their configurations for a multi-container Docker application. In this case, there are two services: Spark-Iceberg and MinIO. Let's break down each section: 1. Spark-Iceberg Service: spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000 Specifies the Docker image to use for the spark-iceberg service. In this case, it uses the tabulario/spark-iceberg:latest image. image: Specifies that the spark-iceberg service depends on the rest and minio services. depends_on: Assigns a specific name (spark-iceberg) to the container. container_name: Sets environment variables for the container, including Spark and AWS credentials. environment: Mounts local directories (./warehouse and ./notebooks) as volumes inside the container. volumes: Maps container ports to host ports for accessing Spark UI and other services. ports: 2. Minio Service: minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"] Specifies the Docker image for the MinIO service. image: Assigns a specific name (MinIO) to the container. container_name: Sets environment variables for configuring MinIO, including root user credentials. environment: Maps container ports to host ports for accessing MinIO UI. ports: Specifies the command to start the MinIO server with specific parameters. command: Another aspect of the MinIO service is , MinIO’s command line tool. mc mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " Specifies that the mc service depends on the MinIO service. depends_on: Specifies the Docker image for the mc service. image: Assigns a specific name (mc) to the container. container_name: Sets environment variables for configuring the MinIO client. environment: Defines the entry point command for the container, including setup steps for the MinIO client. entrypoint: /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      " This sequence of commands essentially performs the following tasks: Removes the existing warehouse directory and its contents from the MinIO server. Creates a new bucket named warehouse. Sets the access policy of the warehouse bucket to public. This Docker Compose file orchestrates a multi-container environment with services for Spark, PostgreSQL, MinIO. It sets up dependencies, environment variables, and commands necessary for running the services together. The services work in tandem to create a development environment for data processing using Spark and Iceberg with MinIO as the object storage backend. Starting Up In a terminal window, cd into the tabular-spark-setup directory in the repository and run the following command: docker-compose up Login into MinIO at with the credentials to see that the warehouse bucket has been created. http://127.0.0.1:9001 admin:password Once all the containers are up and running, you can access your Jupyter Notebook server by navigating to http://localhost:8888 Run one of the sample notebooks and return to MinIO at to see your warehouse populated with data. http://127.0.0.1:9001 Building Your Modern Datalake This tutorial on building a modern datalake with Iceberg, Tabular, and MinIO is just the beginning. This powerful trio opens doors to a world of possibilities. With these tools, you can seamlessly integrate and analyze all your data, structured and unstructured, to uncover hidden patterns and drive data-driven decisions that fuel innovation. Leverage the efficiency and flexibility of this architecture in production to expedite your AI/ML initiatives and unlock the true potential of your machine learning models, accelerating your path to groundbreaking discoveries. Reach out to us at or on our channel if you have any questions as you build. hello@min.io Slack

Building Effective Modern Data Architectures with Iceberg, Tabular and MinIO

আইসবার্গ, ট্যাবুলার এবং মিনিও সহ কার্যকরী আধুনিক ডেটা আর্কিটেকচার তৈরি করা

Erstellen effektiver moderner Datenarchitekturen mit Iceberg, Tabular und MinIO

Creación de arquitecturas de datos modernas y eficaces con Iceberg, Tabular y MinIO

Construire des architectures de données modernes et efficaces avec Iceberg, Tabular et MinIO

आइसबर्ग, टेबुलर और मिनियो के साथ प्रभावी आधुनिक डेटा आर्किटेक्चर का निर्माण

Iceberg、Tabular、MinIO を使用した効果的な最新データ アーキテクチャの構築

Iceberg, Tabular 및 MinIO를 사용하여 효과적인 최신 데이터 아키텍처 구축

Construindo arquiteturas de dados modernas e eficazes com Iceberg, Tabular e MinIO

Создание эффективных современных архитектур данных с помощью Iceberg, Tabular и MinIO

Iceberg, Tabular ve MinIO ile Etkili Modern Veri Mimarileri Oluşturma

Xây dựng kiến trúc dữ liệu hiện đại hiệu quả với Iceberg, Tabular và MinIO

使用 Iceberg、Tabular 和 MinIO 构建有效的现代数据架构

Stories, News & Mentions about DATALAKE

Datalake’s Stories on HackerNoon