What is ETL 💭? is short for , three database functions that are combined into one tool to pull data out of one database and place it into another database. ETL e xtract, t ransform, l oad is the process of from a database. In this stage, the data is collected, often from multiple and different types of sources. Extract reading data is the process of from its previous form into the form it needs to be in so that it can be placed into another database. Transformation occurs by using rules or lookup tables or by combining the data with other data. Transform converting the extracted data is the process of into the target database Load writing the data In the sample apps given by hasura, serverless-etl app is one among them. I will be showing how to implement it using Next.js . This is the to the example given by hasura. link The main features of this app will be :- Add Book to the database From the database data is saved in an algolia index Search for the book in the algolia index How will we implement it ? We will be using graphql mutations to insert book details into the database and when something is inserted into the database an event is triggered which will update the algolia index with the inserted data from the database and this data can be retrieved from the algolia index when searched. That’s it! it is that simple. Pre-requisites: Algolia account Hasura GraphQL Engine (HGE) installation Now that you are ready, Lets begin! Setting up basic Next.js project and Graphql Engine First step is to create a next.js project using . This command will create a basic boilerplate for a next.js project on which we will be working on. yarn create next-app etl-example Add the packages and scripts which are not present in your package.json file from the gist below and install it using . yarn install Next setup a Hasura GraphQL engine using the the deploy option and note down the deployed url and create a file in the root folder with the following code one-click config.js replace the with the url of the deployed hasura engine. Add authentication headers in the config object if you have any. herokuapp-url Creating Book Table We will be having a single table name which will have the details of the book like title and author. For that we have to go to the Hasura GraphQL engine console and create the following table book book table Setting up Algolia Index Next step is to setup an Algolia Index, For that follow the instructions given below. Sign-up for Algolia account Go to the Dashboard, Click on tab on the left sidebar Indices Click on button New Index Name it as demo_serverless_etl_app Goto tab on the sidebar API Keys Copy the (we'll call this ) Application ID **ALGOLIA_APP_ID** Click on tab and then click button All API Keys New API Key Give description as server key Choose the index demo_serverless_etl_app Select under the ACL section Add records Click and copy this API key from the list (we'll call it ) Generate **ALGOLIA_ADMIN_API_KEY** Setting up webhooks For adding the data into the algolia index when some data is inserted into the database, we will be needing a webhook which will be triggered on data insertion and this webhook will add the inserted data into the algolia index. For having a single api endpoint there is no point of having a seperate server. What we can do is setup a node server and add the api endpoint for the webhook and route all other url’s to the next.js handler. For the node server we have to create a file named in the root directory and copy the content from below. The route will be used to trigger an event when data is inserted into the table to add it into the algolia index. Add your algolia APP_ID and the ADMIN_API_KEY in the empty space present in the code. server.js /webhook Building the front-end Next we will setup our page where we will take in the author name and book title which will be added to the database and also a search bar where you can search for any book. Create a file named inside folder and add the following code and also replace the algolia api variables with your values. index.js post Search Component The component provided by package will do the search magic automatically. You do not have to worry about coding all those searching stuff 😉. The component will render the searchbox and component will render the results of your search InstantSearch react-instantsearch-dom SearchBox Hits Deploy Now that wehave the server ready, deploy it anywhere for example and you will get a url for the deployed server note that down, let it be <SERVER_URL>. glitch Creating an Event Trigger Our next step is to create an event trigger on adding a new book to the database, for this go to tab in Hasura Graphql Engine Console and add a new trigger. Replace with Events <TRIGGER_URL> <SERVER_URL>/webhook. Trigger Now you can open <SERVER_URL> in your browser and see the magic! 😉 And that’s it, we have the ETL-demo ready🙌 . I haven’t added any stylings in the code. If you want to make it look more awesome, just add the required css and make it better 😎. Link to the gists :- , , index.js server.js package.json About Me My name is . I am an undergraduate student at the Indian , pursuing my B.Tech degree in Information Technology. Aswin VB Institute of Information Technology, Allahabad I love creating and learning new things which i always do when i have free time. I love to code in J . avaScript and Python You can follow me on and . Twitter GitHub
