In the previous article, I described the concept and design of the Structured Data Service in the Alluxio 2.1.0 release. This article will go through an example to demonstrate how it helps SQL and structured data workloads. Alluxio 2.2.0 is now released since the previous article. I recommend users to update to Alluxio 2.2.0 if trying out this service for the first time. This tutorial requires you have Presto and Hive to be configured together and running. Step 1: Download and Setup Alluxio Download and Deploy Alluxio 2.2.0 and deploy Alluxio on your local computer. Detailed instructions can be found . The following is a summary of the commands mentioned: Download the Alluxio 2.2.0 release here $ tar xf alluxio- .0-bin.tar.gz
$ cd alluxio- .0 # directory corresponds to ${ALLUXIO_HOME}
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
$ echo >> conf/alluxio-site.properties
$ echo >> conf/alluxio-site.properties
$ ./bin/alluxio-mount.sh SudoMount
$ ./bin/alluxio format
$ ./bin/alluxio-start.sh local -f 2.2 2.2 this "alluxio.master.hostname=localhost" "alluxio.master.mount.table.root.ufs=/tmp" Note that no additional configuration is needed to start the new Structured Data Service. Install and Configure the Alluxio Presto Connector The Alluxio Presto Connector is the client for Presto to access Alluxio’s Structured Data Service. In this developer preview version, we need to copy the connector manually to Presto This connector is bundled as part of the Alluxio 2.2.0 release in the directory ${ALLUXIO_HOME}/client/presto/plugins/. Copy the directory corresponding to the Presto version into Presto’s plugin directory. $ cp -R ${ALLUXIO_HOME}/client/presto/plugins/presto-hive-alluxio- / \
${PRESTO_HOME}/plugin/hive-alluxio/ 319 Once the connector is installed, it can be used to configure a Presto catalog. Add a new catalog configuration to Presto by creating the following file $ echo >
${PRESTO_HOME}/etc/catalog/catalog_alluxio.properties "connector.name=hive-alluxio
hive.metastore=alluxio
hive.metastore.alluxio.master.address=localhost:19998" Restart the Presto server for the connector and configuration to take effect. Step 2: Attach a Hive Metastore to Alluxio Catalog Service The Alluxio Catalog Service manages the metadata of structured data components such as databases, tables, and schemas. It also tracks the location of the stored data. This developer preview version supports attaching a Hive Metastore as an UnderDatabase, which is an abstraction of other external catalogs and databases, into the Alluxio Catalog service. To attach the Hive Metastore into the Alluxio Catalog Service, use the command : “attachdb” here $ ./bin/alluxio table attachdb hive thrift: //localhost:9083 hive_db_name Step 3: Use Alluxio Structured Data Management with Presto Once a database is attached, the catalog service can be used from Presto. Start the Presto CLI with the Alluxio catalog: $ presto --catalog catalog_alluxio Any queries run within this CLI will access the Alluxio Catalog Service via the provided connector. The Alluxio Catalog Service will automatically serve the table information from Hive metastore, while transparently using the Alluxio mounted locations. Transform a Table Data transformations is a key benefit of working with structured data in Alluxio, particularly when the underlying files consisting of a table are not stored in a compute-optimized fashion. If the files are in CSV format or the table is split among lots of small files, the Alluxio Transformation Service is able to convert the format to parquet or join multiple small files into larger files. To transform the test table in Hive: $ ./bin/alluxio table transform hive_db_name test_table For more on Data Transformations, see documentations . here Try it out! Alluxio Structured Data Management is an exciting, new effort that provides further benefits for SQL frameworks. with Presto and let us know if you have any feedback for features and issues in the ! On behalf of the entire Alluxio open source community, I invite you to ask questions in our whenever you encounter any issues. Get started with Alluxio Structured Data Service Alluxio Github repository community slack channel

Slack

Join me on Alluxio's Open Source Community! 

2022 - HackerNoon Contributor of the Year - Analytics

2022 - HackerNoon Contributor of the Year - Machine Learning

2022 - HackerNoon Contributor of the Year - Performance

2022 - HackerNoon Contributor of the Year - Sql

Nominated for 2022 - HackerNoon Contributor of the Year - Analytics

Nominated for 2022 - HackerNoon Contributor of the Year - Sql

Nominated for 2022 - HackerNoon Contributor of the Year - Performance

Nominated for 2022 - HackerNoon Contributor of the Year - Machine Learning

Too Long; Didn't Read

Serving Structured Data in Alluxio: Example

Serving Structured Data in Alluxio: Example

Too Long; Didn't Read

Company Mentioned

@bin-fan

RELATED STORIES