Data lakes provide a myriad of benefits. They are data agnostic and don’t require you to define a schema upfront. However, without a proper structure, it may be challenging to find the data that you need. In this article, we’ll address this problem by creating a FastAPI abstraction allowing us to query the AWS Glue metadata catalog. Why FastAPI for a data discovery service? Photo by nine koepfer If we would implement it as a Python package instead, we would assume that everyone knows enough Python to use it. However, your company may have other developers (Java, C#, JavaScript, Go, …) that need access to data from a data lake. By building a REST-ful service for data discovery, you are providing a interface for everyone. This approach has further : programming-language agnostic API-first benefits If you later decide that you want to build a for business users, your task will become much easier since you already have a REST API that provides data as JSON. UI You can combine all the functionality for this purpose in a . single service If you need to add new methods or change the existing ones, you can deploy easier by creating a (ie, api_url/v3) without breaking any workloads that depend on the current version. changes new API version You can to include real-time data streams, unstructured data, as well as to allow querying access logs to identify which data is used most frequently and by which users. extend the functionality We can start by listing all the methods that can be potentially useful for data discovery: listing all databases or , schemas listing all for a specific schema, tables filtering for table names with a specific or , prefix suffix doing a on table names (say, all tables that have anything to do with orders or customers), full-text search doing a on table comments (ex., if we want to find any column related to ), full-text search logistics showing a ( ), full table definition all columns, data types, and column comments showing for a specific table, partitions showing from S3 below a specific partition, all files showing all columns with specific , comment showing all tables that contain a specific , column name finally, the best way for data discovery is to just of a full table. look at the first X rows We have three choices: either using lower-level AWS SDKs, doing queries on Athena’s information_schema, or leveraging awswrangler. In the end, we may combine all of them to satisfy the requirements from the previous section. #1. To list all databases ( ) and tables, we can use the package, especially the module wr.catalog. i.e., schemas awswrangler #2. To filter for table names with a specific or , we can also use wr.catalog.tables. The same is true for retrieving and doing a on those definitions: prefix suffix table definition full-text search #3. List all of a specific table: partitions #4. Show table’s DDL: #5. directory or show specific objects according to file type ( ) and last modified date: List S3 path ex. parquet files #6. Query to find the dataset that you need: column comments #7. Describe Athena : table definition #8. Search for specific : column names The above methods constitute a basic MVP for a data discovery service. It allows querying schemas, tables, columns, column comments ( ), showing a preview of the data, as well as exploring the underlying files, directories, and partitions. aka data dictionary By leveraging services such as AWS X-Ray and Lake Formation, we could add methods to query usage access patterns and identify the most and the least used datasets. If you don’t want to attach your boto3_session separately in each API method, you can attach a global session using: Side note: Implementation: FastAPI MVP The full code for this MVP can be found in . Let’s briefly discuss some of the details. this Github repository In the following demo, we can see all the endpoints applied on a . Brazilian E-commerce dataset from Kaggle The screencast above demonstrates how we can use this API to discover and query e-commerce data stored in an S3 data lake. The end users can identify tables or schemas that they need. By drilling down into the underlying files, data scientists and analysts can explore this data in an interactive notebook or other tools of their choice. Reading a specific parquet, CSV, Excel, or JSON file from S3 with awswrangler is as simple as: How can we serve this API at scale? There are several options to deploy a REST API. If you already use AWS, you may find it useful to leverage serverless services built specifically for building resilient and scalable APIs: and . AWS Lambda API Gateway With the excellent package called mangum, converting our FastAPI to a Lambda handler is as simple as importing this package ( ) and adding a single line of code: handler = Mangum(app=app). from mangum import Mangum The Dockerfile to deploy the API to AWS Lambda looks as follows: We use the official Python Lambda image with Python 3.8 and all required . Then, we copy our remaining API code and specify the that will be used as an entry point to our container. install packages Lambda handler Finally, we have to push the container to ECR — AWS container registry. To replicate this demo, replace 123456789 with your AWS account ID, and adjust your AWS region name. In case you’re wondering: dda is my abbreviation for data-discovery-api. AWS Lambda The container image is deployed to ECR. Now, we need to . We choose the container image option and select our ECR image. create a Lambda function Since querying some tables can take longer than Lambda’s default of just three seconds, we can extend it to three minutes. timeout Finally, we need to attach IAM policies to grant our Lambda function permissions to retrieve data from AWS Glue, Athena, and S3 ( ). of course, we should use more granular permissions for production API Gateway After configuring the Lambda function with our container image, we can create a new API in the API Gateway console. We need to choose REST API and specify a name. Then, we can add a method and resource to configure Lambda Proxy integration. Once all that is set up, we can deploy the API and start serving it. To see it step by step, here is a screencast that demonstrates API Gateway configuration, deployment, and testing the API: How can we monitor our data discovery service? Imagine that you deployed a similar service at your company and it turned out to be a resounding success. Now you need to ensure that it stays , can be easily , and runs . At this point, you can either: healthy maintained without errors Keep an eye on it manually ( ), not recommended Build an yourself by using AWS CloudWatch and related services, observability dashboard Make use of observability tools The implications of each option: With option #1, you will likely go mad. With option #2, you would have to spend extra time building CloudWatch dashboards, jump between several CloudWatch logs and multiple AWS console tabs to fix any potential bugs. With option #3, you only have to sign in to your account  to scans all serverless resources and provides you with visualizations you need to track the health of your APIs and microservices. You can then optionally ( ) build a by combining API Gateway and Lambda resources, and on failure. highly recommended! project dashboard set alerts In the image above, we can see how Dashbird pointed out the task which timed out after 3 seconds ( ). It also reveals when the function experienced a cold start and shows the duration for each API call. default Lambda timeout Lambda For , we can see the average latency, number of API requests, and different types of errors that we encountered in the initial testing of the API. API Gateway Conclusion In this article, we investigated how we can build a REST-ful service for data discovery. By leveraging Python libraries such as FastAPI, awswrangler, boto3, and Mangum, we can build useful APIs in just a few hours rather than weeks. Additionally, by deploying this service to API Gateway and Lambda, we can serve this API at scale with no operational headaches. Lastly, by leveraging Dashbird, you can add observability to your serverless resources without having to install any CloudWatch agents, pulling logs, or building any dashboards. Previously published at https://dashbird.io/blog/data-discovery-api-aws-data-lake/

Discovery

Glue

Is There An Affordable Alternative To Step Functions?

Serverless Playbook For Enterprises

Serverless af

Nominated for 2022 - HackerNoon Contributor of the Year - Cloud Computing

Nominated for 2022 - HackerNoon Contributor of the Year - Devops

Nominated for 2022 - HackerNoon Contributor of the Year - Serverless

Nominated for 2022 - HackerNoon Contributor of the Year - Aws

Too Long; Didn't Read

Make resilience your competitive advantage

How To Use a FastAPI Abstraction To Query AWS Data Lake

How To Use a FastAPI Abstraction To Query AWS Data Lake

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

10 Things in Engineering We Don't Spend Enough Time On

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

10 Things in Engineering We Don't Spend Enough Time On

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

10 Threats to an Open API Ecosystem

10 Indications That You Should Invest in Automation Via APIs

10 Best Practices for Securing Your API

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps