ETL is short for extract, transform, load, three database functions that are combined into one tool to pull data out of one database and place it into another database.
In the sample apps given by hasura, serverless-etl app is one among them. I will be showing how to implement it using Next.js .
This is the link to the example given by hasura.
The main features of this app will be :-
We will be using graphql mutations to insert book details into the database and when something is inserted into the database an event is triggered which will update the algolia index with the inserted data from the database and this data can be retrieved from the algolia index when searched. That’s it! it is that simple.
Now that you are ready, Lets begin!
First step is to create a next.js project using
yarn create next-app etl-example . This command will create a basic boilerplate for a next.js project on which we will be working on.
Add the packages and scripts which are not present in your package.json file from the gist below and install it using
Next setup a Hasura GraphQL engine using the the one-click deploy option and note down the deployed url and create a config.js file in the root folder with the following code
herokuapp-url with the url of the deployed hasura engine. Add authentication headers in the config object if you have any.
We will be having a single table name book which will have the details of the book like title and author. For that we have to go to the Hasura GraphQL engine console and create the following table
Next step is to setup an Algolia Index, For that follow the instructions given below.
Indicestab on the left sidebar
API Keystab on the sidebar
Application ID(we'll call this
All API Keystab and then click
New API Keybutton
Add recordsunder the ACL section
Generateand copy this API key from the list (we'll call it
For adding the data into the algolia index when some data is inserted into the database, we will be needing a webhook which will be triggered on data insertion and this webhook will add the inserted data into the algolia index. For having a single api endpoint there is no point of having a seperate server. What we can do is setup a node server and add the api endpoint for the webhook and route all other url’s to the next.js handler.
For the node server we have to create a file named
server.js in the root directory and copy the content from below. The route
/webhook will be used to trigger an event when data is inserted into the table to add it into the algolia index. Add your algolia APP_ID and the ADMIN_API_KEY in the empty space present in the code.
Next we will setup our page where we will take in the author name and book title which will be added to the database and also a search bar where you can search for any book. Create a file named
post folder and add the following code and also replace the algolia api variables with your values.
The InstantSearch component provided by react-instantsearch-dom package will do the search magic automatically. You do not have to worry about coding all those searching stuff 😉. The SearchBox component will render the searchbox and Hits component will render the results of your search
Now that wehave the server ready, deploy it anywhere for example glitch and you will get a url for the deployed server note that down, let it be <SERVER_URL>.
Our next step is to create an event trigger on adding a new book to the database, for this go to
Events tab in Hasura Graphql Engine Console and add a new trigger. Replace <TRIGGER_URL> with <SERVER_URL>/webhook.
Now you can open <SERVER_URL> in your browser and see the magic! 😉
And that’s it, we have the ETL-demo ready🙌 . I haven’t added any stylings in the code. If you want to make it look more awesome, just add the required css and make it better 😎.
My name is Aswin VB. I am an undergraduate student at the Indian Institute of Information Technology, Allahabad, pursuing my B.Tech degree in Information Technology.