ETL is short for extract, transform, load, three database functions that are combined into one tool to pull data out of one database and place it into another database.
In the sample apps given by hasura, serverless-etl app is one among them. I will be showing how to implement it using Next.js .
This is the link to the example given by hasura.
The main features of this app will be :-
We will be using graphql mutations to insert book details into the database and when something is inserted into the database an event is triggered which will update the algolia index with the inserted data from the database and this data can be retrieved from the algolia index when searched. That’s it! it is that simple.
Now that you are ready, Lets begin!
First step is to create a next.js project using yarn create next-app etl-example
. This command will create a basic boilerplate for a next.js project on which we will be working on.
Add the packages and scripts which are not present in your package.json file from the gist below and install it using yarn install
.
Next setup a Hasura GraphQL engine using the the one-click deploy option and note down the deployed url and create a config.js file in the root folder with the following code
replace the herokuapp-url
with the url of the deployed hasura engine. Add authentication headers in the config object if you have any.
We will be having a single table name book which will have the details of the book like title and author. For that we have to go to the Hasura GraphQL engine console and create the following table
book table
Next step is to setup an Algolia Index, For that follow the instructions given below.
Indices
tab on the left sidebarNew Index
buttondemo_serverless_etl_app
API Keys
tab on the sidebarApplication ID
(we'll call this **ALGOLIA_APP_ID**
)All API Keys
tab and then click New API Key
buttonserver key
demo_serverless_etl_app
indexAdd records
under the ACL sectionGenerate
and copy this API key from the list (we'll call it **ALGOLIA_ADMIN_API_KEY**
)For adding the data into the algolia index when some data is inserted into the database, we will be needing a webhook which will be triggered on data insertion and this webhook will add the inserted data into the algolia index. For having a single api endpoint there is no point of having a seperate server. What we can do is setup a node server and add the api endpoint for the webhook and route all other url’s to the next.js handler.
For the node server we have to create a file named server.js
in the root directory and copy the content from below. The route /webhook
will be used to trigger an event when data is inserted into the table to add it into the algolia index. Add your algolia APP_ID and the ADMIN_API_KEY in the empty space present in the code.
Next we will setup our page where we will take in the author name and book title which will be added to the database and also a search bar where you can search for any book. Create a file named index.js
inside post
folder and add the following code and also replace the algolia api variables with your values.
Search Component
The InstantSearch component provided by react-instantsearch-dom package will do the search magic automatically. You do not have to worry about coding all those searching stuff 😉. The SearchBox component will render the searchbox and Hits component will render the results of your search
Now that wehave the server ready, deploy it anywhere for example glitch and you will get a url for the deployed server note that down, let it be <SERVER_URL>.
Our next step is to create an event trigger on adding a new book to the database, for this go to Events
tab in Hasura Graphql Engine Console and add a new trigger. Replace <TRIGGER_URL> with <SERVER_URL>/webhook.
Trigger
Now you can open <SERVER_URL> in your browser and see the magic! 😉
And that’s it, we have the ETL-demo ready🙌 . I haven’t added any stylings in the code. If you want to make it look more awesome, just add the required css and make it better 😎.
Link to the gists :- index.js , server.js , package.json
My name is Aswin VB. I am an undergraduate student at the Indian Institute of Information Technology, Allahabad, pursuing my B.Tech degree in Information Technology.
I love creating and learning new things which i always do when i have free time. I love to code in JavaScript and Python.