This article describes how engineers in the Data Service Center (DSC) at Tencent PCG (Platform and Content Business Group) leverages Alluxio to optimize the analytics performance and minimize the operating costs in building , a real-time data analytics platform. Tencent Beacon Growing Specifically, by using Alluxio as a distributed caching layer, the HDFS scanning performance for Impala SQL queries is accelerated by 2x out of the box without further optimizations. The original article was published on by guest writer Honghan Tian. Alluxio's Engineering Blog Tencent’s Big Data Analytic App Beacon Growing is an analytics platform that provides analyzers insights of user behaviors in realtime. Beacon comes with both built-in and customized analytic models, such as RETENT, FUNNEL, PATH, etc. Combining the functions of A/B testing and user profiles, developers can select promotional plans more easily. Under the hood, Beacon is powered by a stack of open source technologies. Particularly for the context of this article, we used Impala to query data in Parquet format stored in HDFS. Here is the overview of the software stack currently being deployed by Beacon, to give the readers a better understanding of the system. Challenges Before In Tencent, we build systems to handle data at the Internet-scale. For example, the scan range of a typical query can be 100 billion rows, with hundreds of concurrent queries. To speed up the queries, we have applied optimizations like . In practice, there were still a few issues observed: page index in Parquet files to reduce the I/O cost in scan operations : When more than 100 queries are running concurrently and scanning HDFS, we observed that the usage of host disk was fixed at 100%. I/O Bottleneck under high concurrency :  Insert operations such as “SORT BY” during table creation are very slow due to files merge under HDFS-parquet sort. Note that in our , “SORT BY” helps meet query SLAs without over partitioning the table. Expensive writes application Why Alluxio Based on the above observations, we decided to build a caching layer using SSDs. However, HDFS is not designed to leverage heterogeneous storage. We deployed Alluxio to leverage the speed advantage of SSDs and the capacity advantage of HDDs. The new architecture is shown in the following figure: There are a few changes and improvements observed after integrating Alluxio into Tencent Beacon infrastructure: The range of the average load of HDFS was significantly reduced from  80% ~ 100% down to 50% ~ 70%. The I/O time is also reduced by 50% or more. Better Resource Utilization: : With faster data retrieval, I/O-intensive queries are sped up by 244% with a median improvement of 121% across all queries. Higher Query Performance Alluxio greatly reduces the I/O consumption of hotspot and large queries to the entire HDFS cluster, reducing the overall query failure rate by more than 5% Higher Cluster Stability: Faster execution also reduces the query time in waiting-queue, resulting in 29% reduction in the timeout query failure rate. Reduced Timeout Failure Rate: One of the most valuable features of Alluxio in our use case is . It helps share the same namespace across multiple clusters, making it easier for Impala to access the data resources of multiple HDFS clusters. Unified Namespace The following table is a comparison of some data： Next steps: Locality Groups There are a few on-going projects on the horizon to integrate Alluxio at Tencent PCG. First, we plan to further reduce the operating work by deploying a large Alluxio cluster with worker processes spanning multiple Impala clusters. In other words, Alluxio and Impala processes will be co-located at the same server to reduce network traffic. We are currently experiencing some and working with the core team to address them. issues Second, we want to leverage the Alluxio to further improve Impala performance and eliminate remote reads of Alluxio workers with improved data locality. tiered locality group Specifically, with one but much larger Alluxio cluster deployed, we will divide Alluxio workers nodes into different ‘locality groups’. As a result, when queries read data from different HDFS clusters, data will be cached into the nearest locality group to reduce I/O cost. This feature is an analogy of the in Clickhouse. “layer” configuration” Third, once locality grouping described above is achieved, locally deployed Impala computing groups can be used to elastically expand or shrink computing resources based on different Impala clusters for different business uses. Conclusion In our project “Beacon Growing”, we have deployed Alluxio to improve Impala performance by 2.44x for IO intensive queries and 1.20x for all queries. The query failure rate due to timeout is also reduced by 29%. In the future, we foresee it can reduce disk utilization by over 20% for our planned elastic computing on Impala.

TENCENT

Running Presto Engine in a Hybrid Cloud Architecture

Deep Learning at Alibaba Cloud with Alluxio: How To Run PyTorch on HDFS

Join me on Alluxio's Open Source Community! 

Nominated for 2022 - HackerNoon Contributor of the Year - Analytics

Nominated for 2022 - HackerNoon Contributor of the Year - Sql

Nominated for 2022 - HackerNoon Contributor of the Year - Performance

Nominated for 2022 - HackerNoon Contributor of the Year - Machine Learning

Too Long; Didn't Read

Make resilience your competitive advantage

Accelerating Analytics by 200% with Impala, Alluxio, and HDFS at Tencent

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A New Approach to Solve I/O Challenges in the Machine Learning Pipeline

10 Principles of Proper Database Benchmarking

5 Caching Mechanisms to Speed Up Your Application

A Guide to Paying Less For AWS CloudFront

Accelerate Spark and Hive Jobs on AWS S3 by 10x with Alluxio as a Tiered Storage Solution

A New Approach to Solve I/O Challenges in the Machine Learning Pipeline

10 Principles of Proper Database Benchmarking

5 Caching Mechanisms to Speed Up Your Application

A Guide to Paying Less For AWS CloudFront

Accelerate Spark and Hive Jobs on AWS S3 by 10x with Alluxio as a Tiered Storage Solution

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps