First, the news: ! K8s is a certified environment for Alluxio. Now the take away - . How? Read on. Alluxio support for K8s Helm charts is now available Alluxio brings back data locality for the disaggregated analytics stack in K8s There’s no arguing the rise of containers in real-world deployments over the past few years. Containers simplify running applications in any environment and Kubernetes further transforms the way software and applications are deployed and scaled agnostic of environments. In fact, Kubernetes is increasingly seen as a key technology that enables not only easy resource orchestration in the data center but also in hybrid and multi-cloud environments. While containers and Kubernetes work exceptionally well for stateless applications like web servers and even completely self-contained databases like mongoDB, Couchbase, and others, the stack looks a bit different in the world of advanced analytics and AI. The modern analytical stack is a highly disaggregated stack. Unlike traditional databases or data warehouses, the new stack is split apart. Pick a data lake or two or three to store data (S3, GCS, HDFS, etc.) Pick a computational framework to analyze data (Apache Spark, Presto, Hive, TensorFlow etc.) Make sure all the other dependencies like the catalog service are available (Hive Metastore, AWS Glue, KMS, etc.) Challenges Running the Disaggregated Analytics Stack in K8s Kubernetes greatly simplifies the complexity of deploying so many distributed systems together. And over time, advanced analytics running on K8s clusters will become the norm. But there are still a few critical gaps to make this modern analytical stack effective. Challenge #1 – No Shared Data Access / Caching Layer in the K8s Cluster K8s is a fantastic container orchestration technology and with tools like Helm charts, operators, and more, deployment can be greatly simplified. However, data-intensive workloads like advanced analytics typically need data sharing between jobs to be effective so that data from one job can be easily be accessed by the next job. Without a data access / caching layer, the data needs to be written back to the data lake and then needs to be read back into the K8s cluster again significantly slowing down data pipelines. Challenge #2 – Lost Data Locality With data being stored in S3 or other cloud object stores or on-prem in Hadoop, to perform analytics within the K8s cluster, users have a couple of options. Data needs to either be accessed remotely (meaning poor performance) or needs to be manually copied into the K8s cluster (meaning a lot more additional DevOps and management on a per workload basis). And oftentimes this will carry the burden of managing the differences between those copies which can be hard. The ideal solution is for data locality to be recreated in this disaggregated stack. Challenge #3 – No Data Elasticity for Elastic Compute The beauty of K8s is the flexibility it gets to even the most complex compute workloads – scale up, down, upgrade, restart, etc. based on need and demand. But again the dependency on data being available to compute remains for data-intensive workloads. To scale compute in, out, up, or down, the data within K8s also needs to be able to do the same to leverage the power of the flexibility K8s brings. Data Orchestration can solve these challenges by syncing data into the K8s cluster and allowing for seamless in-memory data access and flexibility to share data across jobs and scale in or out as needed. The News Alluxio has had a docker container for a while, but with Alluxio version 2.1, Kubernetes becomes a first-class environment for Alluxio with advanced testing and certification of K8s. We are now seeing more production deployments with Alluxio and compute frameworks like Presto and Spark in K8s. Also new with Alluxio version 2.1, Alluxio is available for deployment via Helm Charts. What are Helm Charts? Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Charts are easy to create, version, share, and publish — so start using Helm and stop the copy-and-paste. Also Published At: https://www.alluxio.io/blog/kubernetes-alluxio-and-the-disaggregated-analytics-stack/ Written By: Dipti Borkar

Apache

Glue

How to Unify Silos With Data Orchestration

A Primer on Decoupling SQL Engines from Hive Data Warehouse

Join me on Alluxio's Open Source Community! 

Nominated for 2022 - HackerNoon Contributor of the Year - Analytics

Nominated for 2022 - HackerNoon Contributor of the Year - Sql

Nominated for 2022 - HackerNoon Contributor of the Year - Performance

Nominated for 2022 - HackerNoon Contributor of the Year - Machine Learning

Too Long; Didn't Read

Make resilience your competitive advantage

Overcoming Challenges Running the Disaggregated Analytics Stack in K8s

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A New Approach to Solve I/O Challenges in the Machine Learning Pipeline

Advanced Analytics and its Importance in the Healthcare Sector

Enhancing Audit Processes With Advanced Analytical Tools

Inside Red Bull’s Secret Lab Driving Aerospace, Automotive and Clean Energy

10 Great Articles On Data Science And Programming!

A New Approach to Solve I/O Challenges in the Machine Learning Pipeline

Advanced Analytics and its Importance in the Healthcare Sector

Enhancing Audit Processes With Advanced Analytical Tools

Inside Red Bull’s Secret Lab Driving Aerospace, Automotive and Clean Energy

10 Great Articles On Data Science And Programming!

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps