By: Adit Madan, Jasmine Wang and Bin Fan, Alluxio This article highlights the synergy between the two widely adopted open-source projects, and , and demonstrates how together they deliver a . Alluxio Presto self-serve data architecture across clouds What Makes an Architecture Self-serve? Condition 1: Evolution of the data platform does not require changes All data platforms evolve over time, including the addition of a new data store, compute engine, or a new team that needs to access shared data. In either case, a data platform is self-serve if it does not require changes to accommodate evolution. Condition 2: Isolation across Teams Business units don’t step on each other with a self-serve platform. When a new team is introduced, data access by one team should have no impact on the existing usage of the shared data infrastructure. The combination of the above two offers agility, which oftentimes is more important than the cost of physical infrastructure. Data Platform Considerations Below, we introduce some considerations when designing a self-serve platform, and architectural patterns for simple solutions. Consideration 1: Data is shared Between Compute Frameworks There are a large number of specialized compute engines. Each engine is better suited for a specific task, which means there is a need to share data between engines. For example, ETL in a batch processing followed by Presto for interactive queries. Between Different Teams For example, a team is responsible for collection of operational data which is then consumed by multiple other business units. Between Data Centers Across Regions and Cloud Providers This offers the flexibility to choose the most optimal service across environments. The solution for shared data is to have an abstraction layer across heterogeneous compute. Alluxio provides such an abstraction across clouds for seamless sharing of data between Presto and other compute engines regardless of the data store. Consideration 2: Data has ownership domains and processing in place is simple Although replication provides isolation, governance becomes complex as the owner of data enforces strict policies about the consumption of data. Copies introduce redundancy, which is error-prone and has high resource requirements. It may seem obvious that a solution is to not make copies of data, but what about performance when we don’t move data? This calls for a single abstraction layer that takes care of governance, performance, and movement of data across ownership domains. The architecture below shows Presto using the Alluxio layer for access to data regardless of the location. The above design can be broken down into a few simple cases All within a single cloud or a datacenter Shared across multiple data centers or a hybrid cloud In all these cases, the separation of the CONSUMER from the PRODUCER of data is enabled by an abstraction layer that provides more than a simple cache. Advanced preloading and write capabilities guarantee SLAs even with the separation of data from compute. Conclusion: With a self-serve data architecture across clouds, we construct a solution that stands the test of time as a data platform evolves. Learn more from the whitepaper , and see how companies including Facebook, TikTok, Electronic Arts, Walmart, Tencent, Comcast, etc level up their current Presto platform leveraging Alluxio. Presto with Alluxio Overview – Architecture Evolution for Interactive Queries First Published here

Comcast

Facebook

TENCENT

Modernize Your Spark Platform with Data Orchestration

Best Practices for Apache Ranger Based Authorization for Your Data Platform

Join me on Alluxio's Open Source Community! 

Nominated for 2022 - HackerNoon Contributor of the Year - Analytics

Nominated for 2022 - HackerNoon Contributor of the Year - Sql

Nominated for 2022 - HackerNoon Contributor of the Year - Performance

Nominated for 2022 - HackerNoon Contributor of the Year - Machine Learning

Too Long; Didn't Read

Make resilience your competitive advantage

Using a Self-serve Data Architecture to Share Data Across Development Teams

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A New Approach to Solve I/O Challenges in the Machine Learning Pipeline

10 Ways to Future-Proof Your Business With Cloud

10 Upcoming DevOps Conferences for 2018

101 Stories To Learn About Cloud Infrastructure

The Ten Top Cities for Highest Cloud Engineering Salaries

A New Approach to Solve I/O Challenges in the Machine Learning Pipeline

10 Ways to Future-Proof Your Business With Cloud

10 Upcoming DevOps Conferences for 2018

101 Stories To Learn About Cloud Infrastructure

The Ten Top Cities for Highest Cloud Engineering Salaries

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps