How To Optimize Large S3 API Costs using Alluxio

Written by bin-fan | Published 2020/08/06
Tech Story Tags: benchmark | hadoop | cloud-storage | sql | apache-spark | distributed-systems | open-source | aws

TLDR This article describes how engineers at Datasapiens brought down S3 API costs by implementing Alluxio as a data orchestration layer between S3 and Presto. The main conclusion from these use cases is that using AlluxIO has the following benefits: lower latency in data processing pipelines. The infrastructure costs were above expectations due to S3 cost increases, making up a large proportion of the overall cost (described in detail below) This architecture achieves faster performance and reduced compilation times. In addition, we saw a drop in infrastructure costs due to the increase in S3 costs.via the TL;DR App

no story

Written by bin-fan | VP of Open Source and Founding Member @Alluxio
Published by HackerNoon on 2020/08/06