Building a Large-Scale Interactive SQL Query Engine with Open Source Softwareby@bin-fan
943 reads

Building a Large-Scale Interactive SQL Query Engine with Open Source Software

March 7th 2020
4 min
by @bin-fan 943 reads
tldt arrow
Read on Terminal Reader🖨️

Too Long; Didn't Read

JD.com is running a data platform with more than 40,000 servers, running more than 1 million jobs per day, managing over 650PB of data. This article describes how JD built an interactive OLAP platform combining two open-source technologies: Presto and Alluxio. The enormous scale caused issues in achieving good data locality, which significantly impacts the performance of jobs running on Presto when reading from HDFS. The platform is over-utilized to the point where YARN is unable to schedule Presto jobs on its local HDFS datanode.

Company Mentioned

Mention Thumbnail
featured image - Building a Large-Scale Interactive SQL Query Engine with Open Source Software
Bin Fan HackerNoon profile picture

@bin-fan

Bin Fan

About @bin-fan
LEARN MORE ABOUT @BIN-FAN'S EXPERTISE AND PLACE ON THE INTERNET.
react to story with heart

RELATED STORIES

L O A D I N G
. . . comments & more!
Hackernoon hq - po box 2206, edwards, colorado 81632, usa