Create a Full Autocomplete Search Application with Elasticsearch, Kibana, NestJS and React

[Elasticsearch](https://en.wikipedia.org/wiki/Elasticsearch) is a distributed, RESTful search and analytics engine that can handle an expanding range of use cases. It offers a full-text search engine with a multitenant capability, an HTTP web interface, and schema-free JSON documents, all with simple installation.

\ > Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

\ In this article, I will be walking you through how to set up elasticsearch on your PC.

\ ## How to setup, install, and configure Elasticsearch for Mac

In order to download Elasticsearch, go to the [Elasticsearch Download Page](https://www.elastic.co/downloads/elasticsearch), select the respective operating system and click on the download button as shown below:

![Download elasticsearch](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.553Z-cl81znskh00010as69k54a4up)

\ After downloading, extract the compressed file using the command below:

\ ```bash $ tar -xzvf elasticsearch-8.4.1-darwin-aarch64.tar.gz ```

\ ### Configuring Elasticsearch

After extracting elasticsearch, depending on your use case, you may want to configure elasticsearch to suit you. You can take a look at the config file using the command below:

\ ``` $ vi elasticsearch-8.4.1/config/elasticsearch.yml ```

\ **Cluster:** You can have multiple nodes in your cluster, in order to do this, you need to ensure that the cluster name on your nodes matches.

\ ``` # -------------------------------- Cluster ---------------------------------- # # Use a descriptive name for your cluster: # cluster.name: air-elastic # ```

\ **Node:** This is an identifier for the individual nodes in your cluster

\ ``` # -------------------------------- Node ---------------------------------- # # Use a descriptive name for your cluster: # node.name: airscholar-node # ```

\ For other configurations, check `elasticsearch.yml` to fine-tune the configs to suit your use case.

\ ## Starting up Elasticsearch

In order to start up elasticsearch engine use the command below:

\ ```bash $ bin/elasticsearch ```

\ When starting up `elasticsearch` for the first time, you will be presented with a password that needs to be changed:

\ ![elastic-password-change](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.554Z-cl81znskj00020as66grl6ymn)

\ ### Changing your password for the first time

You should change your password after starting up the elasticsearch for the first time, of course, you can do that using this command:

\ ``` bin/elasticsearch-reset-password -u elastic ```

\

:::tip NOTE: ENSURE YOU KEEP YOUR ELASTIC PASSWORD SAFE, IT WILL BE REQUIRED IN ORDER TO CONNECT TO ELASTICSEARCH

:::

\ ### Testing Elasticsearch

In your browser, goto `https://localhost:9200` or the specified hostname and port in your `elasticonfig.yml`. You will be asked for `username` (elastic) and `password` (the password you changed to when starting elasticsearch for the first time)

\ ![elastic-authentication](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.556Z-cl81znskk00030as6hmuyfsee)

\ You should see something like this:

\ ```json { "name" : "airscholar-node", "cluster_name" : "air-elastic", "cluster_uuid" : "7g3LmMYFRmyxTyR6ZI-b7A", "version" : { "number" : "8.4.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c", "build_date" : "2022-08-26T12:11:43.232597118Z", "build_snapshot" : false, "lucene_version" : "9.3.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } ```

\ If you have any challenges setting up elasticsearch, feel free to drop a comment, and I will try to respond. You can also check the [official elasticsearch documentation](https://www.elastic.co/guide/index.html) for more detailed information about elasticsearch.

\ ### Kibana

> Kibana is a source-available data visualization dashboard software for Elasticsearch, whose free and open source successor in OpenSearch is OpenSearch Dashboards.

\ ### How does Kibana help us in this case you may ask?

[Kibana](https://en.wikipedia.org/wiki/Kibana) is a great visualization tool that helps us with our elasticsearch queries and manipulation. It is part of B.E.L.K (Beats, Elasticsearch, Logstash, and Kibana) stack. In case you don't know much about it, follow along and you will be just fine! (A separate article on Kibana can be written as an addendum to this series if you want, let me know in the comment section!)

\ ## Setting up and Configuring Kibana

In order to download Kibana, go to [Kibana Download Page](https://www.elastic.co/downloads/kibana), and click on the download button.

![Download Kibana](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.557Z-cl81znskl00040as666f9bjzf)

\ After downloading, extract the zip file using this command:

\ ``` $ tar -xzvf kibana-8.4.1-darwin-aarch64.tar.gz $ cd kibana-8.4.1 ```

\ There are two ways to configure and connect Kibana to elasticsearch, one way is to manually input the required keys into `kibana.yml` (in `config/kibana.yml`) or run Kibana and configure it on the UI. I'm choosing the latter 😉!

\ Run kibana using this command:

\ ``` bin/kibana ```

\ When starting Kibana for the first time, you will be presented with this screen:

\ ![kibana enrolment](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.644Z-cl81znsn0000n0as6cbaohgk3)

\ There are two ways to tackle this;

1. Use the elastic enrollment token (usually generated on your terminal when starting elastic for the first time) see below;

![elastic enrolment](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.643Z-cl81znsmz000m0as604y956n0)

\ Copy and paste the enrollment token above to Kibana and click `Configure Elastic`. You will be fine.

\

2. Manually configure Kibana with elastic using the elastic URL. Click on check address and click `Configure elastic`.

\ ![elastic URL](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.642Z-cl81znsmy000l0as60ypa9fux)

\ Enter `kibana_system` password generated. Or click forgot password (if you can not recall the password), and paste the command below to your terminal to reset the password.

\ ``` bin/elasticsearch-reset-password --username kibana_system ```

\ You will be asked to enter a verification code and check your terminal (where you are running kibana to retrieve it)

\ ![verification code](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.635Z-cl81znsmr000k0as6eiee8n8x)

\ That's it! You are all set up.

\ Enter your elastic username and password to log in to kibana and you are in!

\ ![elastic login](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.635Z-cl81znsmr000j0as6c0ak6x68)

\ ### Querying elastic using Kibana

Let's run a sample query to test our connection, shall we?

1. Click on the hamburger (three lines on the top left corner of your screen) 2. Scroll down to the bottom of the page 3. Click on `Dev tools`

\ You should see something like this:

![dev tools](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.634Z-cl81znsmq000i0as6as697s9a)

\ On the left pane is where queries are written, and the right pane is where the outputs are presented.

\ Ensure you click on any part of the query and select the play button.

\ The output of the query will be presented on the right pane.

\ ### Loading data into elasticsearch

To enable us to write our code effectively, we need data loaded into our `elasticsearch`. We will be using a sample dataset from Kaggle ([Download it here](https://drive.google.com/file/d/1ZtcGb59EkLN0k2q4cpYS3XwYXwQdt4t_/view?usp=sharing)).

\ Follow the FOUR steps below to load it up into elasticsearch:

\

1. Open up Kibana ( )

\ ![Kibana upload](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.633Z-cl81znsmp000h0as6h2ohh44l)

\ 2. Under `Get started by adding integrations` Click on Upload a file:

\ ![upload file](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.627Z-cl81znsmj000g0as66kyx6tvu)

\ 3. Click on import and enter the name of the index you want to put the data in.

\ ![index name](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.626Z-cl81znsmi000f0as62l5c4ueq)

\ 4. Click on import (final page)

\

![import](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.625Z-cl81znsmh000e0as640eodeoz)

\ If you made it to this point, you have successfully imported data into elasticsearch.

\ ### Querying for sample

Goto DevTools (Hamburger on the top left corner of the screen > Scroll down to Management > DevTools)

\ Run the query below (select it and click on the play button)

\ ``` GET tmdb_movies/_search ```

\ If you see this, we are good to go!

\ ![querying elasticsearch](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.624Z-cl81znsmg000d0as682uo9qcr)

---

\ Now, Let's dive into coding, shall we 😊?

## NestJS

> NestJS is a progressive Node. js framework that helps build server-side applications. Nest extends Node. js frameworks like Express or Fastify adding modular organization and a wide range of other libraries to take care of repetitive tasks. It's open-source, uses TypeScript, and is a very versatile Node.

\ ### Creating a NestJs Application

Run the command below to install nestcli and create a new NestJs Application (in the article, the name of the app will be nest-elastic).

\ ```bash $ npm i -g @nestjs/cli $ nest new nest-elastic ```

\ You will be asked to select a package manager, you can select npm, yarn, or pnpm. I will be selecting yarn (you can choose any other one you want 😉). Your project will be set up and we should be ready to code!

\ ![project setup](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.578Z-cl81znsl6000c0as652kcd2ko)

\ ### Adding elasticsearch to your app

Run the command below to add elasticsearch to your `nest-elastic` and other dependencies:

\ ```bash yarn add @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config ```

\ In your root folder add your `.env` file with the following contents:

\ ``` ELASTICSEARCH_NODE=https://localhost:9200 ELASTICSEARCH_USERNAME=elastic ELASTICSEARCH_PASSWORD=elasticPasswordGoesHere ELASTICSEARCH_MAX_RETRIES=10 ELASTICSEARCH_REQ_TIMEOUT=50000 ELASTICSEARCH_INDEX=tmdb_movies ```

\ Let's create a separate module that handles only search using elasticsearch. A simple shortcut is to use the command below (you are welcome to do it manually if you want to):

\ ``` nest g resource search ```

\ \ ![added search module](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.577Z-cl81znsl5000b0as61zvd647i)

\ Update the `search.module.ts` to have the content below:

\ ```typescript import { Module } from '@nestjs/common'; import { SearchService } from './search.service'; import { SearchController } from './search.controller'; import { ConfigModule, ConfigService } from '@nestjs/config'; import { ElasticsearchModule } from '@nestjs/elasticsearch';

@Module({ imports: [ ConfigModule, ElasticsearchModule.registerAsync({ imports: [ConfigModule], useFactory: async (configService: ConfigService) => ({ node: configService.get('ELASTICSEARCH_NODE'), auth: { username: configService.get('ELASTICSEARCH_USERNAME'), password: configService.get('ELASTICSEARCH_PASSWORD'), }, maxRetries: configService.get('ELASTICSEARCH_MAX_RETRIES'), requestTimeout: configService.get('ELASTICSEARCH_REQ_TIMEOUT'), }), inject: [ConfigService], }), ], controllers: [SearchController], providers: [SearchService], exports: [SearchService], }) export class SearchModule {} ```

\ Update `search.service.ts` with the content below:

\ ```typescript import { ConfigService } from '@nestjs/config'; import { Injectable } from '@nestjs/common'; import { ElasticsearchService } from '@nestjs/elasticsearch';

type dataResponse = { UnitPrice: number; Description: string; Quantity: number; Country: string; InvoiceNo: string; InvoiceDate: Date; CustomerID: number; StockCode: string; };

@Injectable() export class SearchService { constructor( private readonly esService: ElasticsearchService, private readonly configService: ConfigService, ) {}

async search(search: {key: string}) { let results = new Set(); const response = await this.esService.search({ index: this.configService.get('ELASTICSEARCH_INDEX'), body: { size: 50, query: { match_phrase: search }, }, }); const hits = response.hits.hits; hits.map((item) => { results.add(item._source as dataResponse); });

return { results: Array.from(results), total: response.hits.total }; } } ```

---

\ Now, let's add movies modules:

\ ``` nest g resource movies ```

\ Update `movies.controller.ts` with the content below:

\ ```typescript import { SearchService } from './../search/search.service'; import { Body, Controller, Post } from '@nestjs/common';

@Controller('movies') export class MoviesController { constructor(private readonly searchService: SearchService) {}

@Post('search') async search(@Body() body) { return await this.searchService.search(body.data); } } ```

\ Then `movies.module.ts`

\ ```typescript import { SearchModule } from './../search/search.module'; import { Module } from '@nestjs/common'; import { MoviesService } from './movies.service'; import { MoviesController } from './movies.controller';

@Module({ imports: [SearchModule], controllers: [MoviesController], providers: [MoviesService], }) export class MoviesModule {} ```

\ Finally, update `app.module.ts`

\ ```typescript import { MoviesModule } from './movies/movies.module'; import { ConfigModule } from '@nestjs/config'; import { Module } from '@nestjs/common'; import { AppController } from './app.controller'; import { AppService } from './app.service'; import { SearchModule } from './search/search.module';

@Module({ imports: [MoviesModule, ConfigModule.forRoot(), SearchModule], controllers: [AppController], providers: [AppService], }) export class AppModule {} ```

\ Your `package.json` should look like this:

\ ```json { "name": "nest-elastic", "version": "0.0.1", "description": "", "author": "Yusuf Ganiyu", "private": true, "license": "UNLICENSED", "scripts": { "prebuild": "rimraf dist", "build": "nest build", "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"", "start": "nest start", "start:dev": "nest start --watch", "start:debug": "nest start --debug --watch", "start:prod": "node dist/main", "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix", "test": "jest", "test:watch": "jest --watch", "test:cov": "jest --coverage", "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand", "test:e2e": "jest --config ./test/jest-e2e.json" }, "dependencies": { "@elastic/elasticsearch": "^8.4.0", "@nestjs/common": "^9.0.0", "@nestjs/config": "^2.2.0", "@nestjs/core": "^9.0.0", "@nestjs/elasticsearch": "^9.0.0", "@nestjs/mapped-types": "*", "@nestjs/platform-express": "^9.0.0", "reflect-metadata": "^0.1.13", "rimraf": "^3.0.2", "rxjs": "^7.2.0" }, "devDependencies": { "@nestjs/cli": "^9.0.0", "@nestjs/schematics": "^9.0.0", "@nestjs/testing": "^9.0.0", "@types/express": "^4.17.13", "@types/jest": "28.1.4", "@types/node": "^16.0.0", "@types/supertest": "^2.0.11", "@typescript-eslint/eslint-plugin": "^5.0.0", "@typescript-eslint/parser": "^5.0.0", "eslint": "^8.0.1", "eslint-config-prettier": "^8.3.0", "eslint-plugin-prettier": "^4.0.0", "jest": "28.1.2", "prettier": "^2.3.2", "source-map-support": "^0.5.20", "supertest": "^6.1.3", "ts-jest": "28.0.5", "ts-loader": "^9.2.3", "ts-node": "^10.0.0", "tsconfig-paths": "4.0.0", "typescript": "^4.3.5" }, "jest": { "moduleFileExtensions": [ "js", "json", "ts" ], "rootDir": "src", "testRegex": ".*\\.spec\\.ts$", "transform": { "^.+\\.(t|j)s$": "ts-jest" }, "collectCoverageFrom": [ "**/*.(t|j)s" ], "coverageDirectory": "../coverage", "testEnvironment": "node" } } ```

\ ### Running the app

You can fire up the app in the dev environment using `yarn start:dev`

\ ![running terminal](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.576Z-cl81znsl4000a0as6b5c0cssy)

\ ### Testing

![postman -1](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.575Z-cl81znsl300090as6h0b7gz1p)

\ ![postman -2](https://cdn.hackernoon.com/images/JaI04JxjP5WiiJKlOeLrImL5dAw2-2022-09-14T19:01:07.568Z-cl81znskx00080as63prx27ld)

\ You can access the full [source code here](https://github.com/airscholar/nest-elastic-backend).

\ ## Setting up a react project

You can set up a simple react app using this command (or check out this [detailed react documentation](https://reactjs.org/docs/create-a-new-react-app.html)):

\ ``` $ npx create-react-app nest-elastic-frontend ```

\ Once your app is set up, open it in your favorite IDE, mine is VSCode.

\ We will need to install `axios` as a dependency. If you prefer `npm` package manager, you will have to run `npm install axios` but if you preferred yarn, `yarn add axios`.

\ ### We need to update three files

1. `public/index.html`

\ You need to add bootstrap CDN. (PS: I have removed the comments to reduce the length of the file).

\ ```markup React App You need to enable JavaScript to run this app. ```

\

2. `src/App.js`

\

```javascript import './App.css'; import axios from 'axios'; import { useState } from 'react'; const App = () => { const [searchResponse, setSearchResponse] = useState([]); const [totalValue, setTotalValue] = useState();

const handleChange = async e => { const { data } = await axios.post('http://localhost:8000/movies/search', { data: { title: e.target.value, }, });

setSearchResponse(data.results); setTotalValue(data.total.value); }; return ( Search All Fields {totalValue ?? 0} {totalValue > 1 ? 'Records' : 'Record'} Found Title Overview Revenue:Budget ($) {searchResponse.map((res, idx) => ( {res.title} {res.overview}

"{res.tagline}" {res.revenue.toLocaleString()}:{res.budget.toLocaleString()}

))} ); };

export default App; ```

\

3. Lastly, `src/index.css`

\

```javascript body { margin: 0; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; }

code { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }

.search-table { padding: 10%; margin-top: -6%; } .search-box { background: #c1c1c1; border: 1px solid #ababab; padding: 3%; } .search-box input:focus { box-shadow: none; border: 2px solid #eeeeee; } .search-list { background: #fff; border: 1px solid #ababab; border-top: none; } .search-list h3 { background: #eee; padding: 3%; margin-bottom: 0%; }