Create a Full Autocomplete Search Application with Elasticsearch, Kibana, NestJS and React

is a distributed, RESTful search and analytics engine that can handle an expanding range of use cases. It offers a full-text search engine with a multitenant capability, an HTTP web interface, and schema-free JSON documents, all with simple installation. Elasticsearch Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. In this article, I will be walking you through how to set up elasticsearch on your PC. How to setup, install, and configure Elasticsearch for Mac In order to download Elasticsearch, go to the , select the respective operating system and click on the download button as shown below: Elasticsearch Download Page After downloading, extract the compressed file using the command below: $ tar -xzvf elasticsearch-8.4.1-darwin-aarch64.tar.gz Configuring Elasticsearch After extracting elasticsearch, depending on your use case, you may want to configure elasticsearch to suit you. You can take a look at the config file using the command below: $ vi elasticsearch-8.4.1/config/elasticsearch.yml You can have multiple nodes in your cluster, in order to do this, you need to ensure that the cluster name on your nodes matches. Cluster: # -------------------------------- Cluster ---------------------------------- # # Use a descriptive name for your cluster: # cluster.name: air-elastic # This is an identifier for the individual nodes in your cluster Node: # -------------------------------- Node ---------------------------------- # # Use a descriptive name for your cluster: # node.name: airscholar-node # For other configurations, check to fine-tune the configs to suit your use case. elasticsearch.yml Starting up Elasticsearch In order to start up elasticsearch engine use the command below: $ bin/elasticsearch When starting up for the first time, you will be presented with a password that needs to be changed: elasticsearch Changing your password for the first time You should change your password after starting up the elasticsearch for the first time, of course, you can do that using this command: bin/elasticsearch-reset-password -u elastic NOTE: ENSURE YOU KEEP YOUR ELASTIC PASSWORD SAFE, IT WILL BE REQUIRED IN ORDER TO CONNECT TO ELASTICSEARCH Testing Elasticsearch In your browser, goto or the specified hostname and port in your . You will be asked for (elastic) and (the password you changed to when starting elasticsearch for the first time) https://localhost:9200 elasticonfig.yml username password You should see something like this: { "name" : "airscholar-node", "cluster_name" : "air-elastic", "cluster_uuid" : "7g3LmMYFRmyxTyR6ZI-b7A", "version" : { "number" : "8.4.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c", "build_date" : "2022-08-26T12:11:43.232597118Z", "build_snapshot" : false, "lucene_version" : "9.3.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } If you have any challenges setting up elasticsearch, feel free to drop a comment, and I will try to respond. You can also check the for more detailed information about elasticsearch. official elasticsearch documentation Kibana Kibana is a source-available data visualization dashboard software for Elasticsearch, whose free and open source successor in OpenSearch is OpenSearch Dashboards. How does Kibana help us in this case you may ask? is a great visualization tool that helps us with our elasticsearch queries and manipulation. It is part of B.E.L.K (Beats, Elasticsearch, Logstash, and Kibana) stack. In case you don't know much about it, follow along and you will be just fine! (A separate article on Kibana can be written as an addendum to this series if you want, let me know in the comment section!) Kibana Setting up and Configuring Kibana In order to download Kibana, go to , and click on the download button. Kibana Download Page After downloading, extract the zip file using this command: $ tar -xzvf kibana-8.4.1-darwin-aarch64.tar.gz $ cd kibana-8.4.1 There are two ways to configure and connect Kibana to elasticsearch, one way is to manually input the required keys into (in ) or run Kibana and configure it on the UI. I'm choosing the latter 😉! kibana.yml config/kibana.yml Run kibana using this command: bin/kibana When starting Kibana for the first time, you will be presented with this screen: There are two ways to tackle this; Use the elastic enrollment token (usually generated on your terminal when starting elastic for the first time) see below; Copy and paste the enrollment token above to Kibana and click . You will be fine. Configure Elastic Manually configure Kibana with elastic using the elastic URL. Click on check address and click . Configure elastic Enter password generated. Or click forgot password (if you can not recall the password), and paste the command below to your terminal to reset the password. kibana_system bin/elasticsearch-reset-password --username kibana_system You will be asked to enter a verification code and check your terminal (where you are running kibana to retrieve it) That's it! You are all set up. Enter your elastic username and password to log in to kibana and you are in! Querying elastic using Kibana Let's run a sample query to test our connection, shall we? Click on the hamburger (three lines on the top left corner of your screen) Scroll down to the bottom of the page Click on Dev tools You should see something like this: On the left pane is where queries are written, and the right pane is where the outputs are presented. Ensure you click on any part of the query and select the play button. The output of the query will be presented on the right pane. Loading data into elasticsearch To enable us to write our code effectively, we need data loaded into our . We will be using a sample dataset from Kaggle ( ). elasticsearch Download it here Follow the FOUR steps below to load it up into elasticsearch: Open up Kibana ( ) http://localhost:5601 Under Click on Upload a file: Get started by adding integrations Click on import and enter the name of the index you want to put the data in. Click on import (final page) If you made it to this point, you have successfully imported data into elasticsearch. Querying for sample Goto DevTools (Hamburger on the top left corner of the screen > Scroll down to Management > DevTools) Run the query below (select it and click on the play button) GET tmdb_movies/_search If you see this, we are good to go! Now, Let's dive into coding, shall we 😊? NestJS NestJS is a progressive Node. js framework that helps build server-side applications. Nest extends Node. js frameworks like Express or Fastify adding modular organization and a wide range of other libraries to take care of repetitive tasks. It's open-source, uses TypeScript, and is a very versatile Node. Creating a NestJs Application Run the command below to install nestcli and create a new NestJs Application (in the article, the name of the app will be nest-elastic). $ npm i -g @nestjs/cli $ nest new nest-elastic You will be asked to select a package manager, you can select npm, yarn, or pnpm. I will be selecting yarn (you can choose any other one you want 😉). Your project will be set up and we should be ready to code! Adding elasticsearch to your app Run the command below to add elasticsearch to your and other dependencies: nest-elastic yarn add @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config In your root folder add your file with the following contents: .env ELASTICSEARCH_NODE=https://localhost:9200 ELASTICSEARCH_USERNAME=elastic ELASTICSEARCH_PASSWORD=elasticPasswordGoesHere ELASTICSEARCH_MAX_RETRIES=10 ELASTICSEARCH_REQ_TIMEOUT=50000 ELASTICSEARCH_INDEX=tmdb_movies Let's create a separate module that handles only search using elasticsearch. A simple shortcut is to use the command below (you are welcome to do it manually if you want to): nest g resource search Update the to have the content below: search.module.ts import { Module } from '@nestjs/common'; import { SearchService } from './search.service'; import { SearchController } from './search.controller'; import { ConfigModule, ConfigService } from '@nestjs/config'; import { ElasticsearchModule } from '@nestjs/elasticsearch';

@Module({ imports: [ ConfigModule, ElasticsearchModule.registerAsync({ imports: [ConfigModule], useFactory: async (configService: ConfigService) => ({ node: configService.get('ELASTICSEARCH_NODE'), auth: { username: configService.get('ELASTICSEARCH_USERNAME'), password: configService.get('ELASTICSEARCH_PASSWORD'), }, maxRetries: configService.get('ELASTICSEARCH_MAX_RETRIES'), requestTimeout: configService.get('ELASTICSEARCH_REQ_TIMEOUT'), }), inject: [ConfigService], }), ], controllers: [SearchController], providers: [SearchService], exports: [SearchService], }) export class SearchModule {} Update with the content below: search.service.ts import { ConfigService } from '@nestjs/config'; import { Injectable } from '@nestjs/common'; import { ElasticsearchService } from '@nestjs/elasticsearch';

type dataResponse = { UnitPrice: number; Description: string; Quantity: number; Country: string; InvoiceNo: string; InvoiceDate: Date; CustomerID: number; StockCode: string; };

@Injectable() export class SearchService { constructor( private readonly esService: ElasticsearchService, private readonly configService: ConfigService, ) {}

async search(search: {key: string}) { let results = new Set(); const response = await this.esService.search({ index: this.configService.get('ELASTICSEARCH_INDEX'), body: { size: 50, query: { match_phrase: search }, }, }); const hits = response.hits.hits; hits.map((item) => { results.add(item._source as dataResponse); });

return { results: Array.from(results), total: response.hits.total }; } } Now, let's add movies modules: nest g resource movies Update with the content below: movies.controller.ts import { SearchService } from './../search/search.service'; import { Body, Controller, Post } from '@nestjs/common';

@Controller('movies') export class MoviesController { constructor(private readonly searchService: SearchService) {}

@Post('search') async search(@Body() body) { return await this.searchService.search(body.data); } } Then movies.module.ts import { SearchModule } from './../search/search.module'; import { Module } from '@nestjs/common'; import { MoviesService } from './movies.service'; import { MoviesController } from './movies.controller';

@Module({ imports: [SearchModule], controllers: [MoviesController], providers: [MoviesService], }) export class MoviesModule {} Finally, update app.module.ts import { MoviesModule } from './movies/movies.module'; import { ConfigModule } from '@nestjs/config'; import { Module } from '@nestjs/common'; import { AppController } from './app.controller'; import { AppService } from './app.service'; import { SearchModule } from './search/search.module';

@Module({ imports: [MoviesModule, ConfigModule.forRoot(), SearchModule], controllers: [AppController], providers: [AppService], }) export class AppModule {} Your should look like this: package.json { "name": "nest-elastic", "version": "0.0.1", "description": "", "author": "Yusuf Ganiyu", "private": true, "license": "UNLICENSED", "scripts": { "prebuild": "rimraf dist", "build": "nest build", "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"", "start": "nest start", "start:dev": "nest start --watch", "start:debug": "nest start --debug --watch", "start:prod": "node dist/main", "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix", "test": "jest", "test:watch": "jest --watch", "test:cov": "jest --coverage", "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand", "test:e2e": "jest --config ./test/jest-e2e.json" }, "dependencies": { "@elastic/elasticsearch": "^8.4.0", "@nestjs/common": "^9.0.0", "@nestjs/config": "^2.2.0", "@nestjs/core": "^9.0.0", "@nestjs/elasticsearch": "^9.0.0", "@nestjs/mapped-types": "*", "@nestjs/platform-express": "^9.0.0", "reflect-metadata": "^0.1.13", "rimraf": "^3.0.2", "rxjs": "^7.2.0" }, "devDependencies": { "@nestjs/cli": "^9.0.0", "@nestjs/schematics": "^9.0.0", "@nestjs/testing": "^9.0.0", "@types/express": "^4.17.13", "@types/jest": "28.1.4", "@types/node": "^16.0.0", "@types/supertest": "^2.0.11", "@typescript-eslint/eslint-plugin": "^5.0.0", "@typescript-eslint/parser": "^5.0.0", "eslint": "^8.0.1", "eslint-config-prettier": "^8.3.0", "eslint-plugin-prettier": "^4.0.0", "jest": "28.1.2", "prettier": "^2.3.2", "source-map-support": "^0.5.20", "supertest": "^6.1.3", "ts-jest": "28.0.5", "ts-loader": "^9.2.3", "ts-node": "^10.0.0", "tsconfig-paths": "4.0.0", "typescript": "^4.3.5" }, "jest": { "moduleFileExtensions": [ "js", "json", "ts" ], "rootDir": "src", "testRegex": ".*\\.spec\\.ts$", "transform": { "^.+\\.(t|j)s$": "ts-jest" }, "collectCoverageFrom": [ "**/*.(t|j)s" ], "coverageDirectory": "../coverage", "testEnvironment": "node" } } Running the app You can fire up the app in the dev environment using yarn start:dev Testing You can access the full . source code here Setting up a react project You can set up a simple react app using this command (or check out this ): detailed react documentation $ npx create-react-app nest-elastic-frontend Once your app is set up, open it in your favorite IDE, mine is VSCode. We will need to install as a dependency. If you prefer package manager, you will have to run but if you preferred yarn, . axios npm npm install axios yarn add axios We need to update three files public/index.html You need to add bootstrap CDN. (PS: I have removed the comments to reduce the length of the file). React App You need to enable JavaScript to run this app. src/App.js import './App.css'; import axios from 'axios'; import { useState } from 'react'; const App = () => { const [searchResponse, setSearchResponse] = useState([]); const [totalValue, setTotalValue] = useState();

const handleChange = async e => { const { data } = await axios.post('http://localhost:8000/movies/search', { data: { title: e.target.value, }, });

setSearchResponse(data.results); setTotalValue(data.total.value); }; return ( Search All Fields {totalValue ?? 0} {totalValue > 1 ? 'Records' : 'Record'} Found Title Overview Revenue:Budget ($) {searchResponse.map((res, idx) => ( {res.title} {res.overview}

"{res.tagline}" {res.revenue.toLocaleString()}:{res.budget.toLocaleString()}

))} ); };

export default App; Lastly, src/index.css body { margin: 0; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; }

code { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; }

.search-table { padding: 10%; margin-top: -6%; } .search-box { background: #c1c1c1; border: 1px solid #ababab; padding: 3%; } .search-box input:focus { box-shadow: none; border: 2px solid #eeeeee; } .search-list { background: #fff; border: 1px solid #ababab; border-top: none; } .search-list h3 { background: #eee; padding: 3%; margin-bottom: 0%; }

.title { word-wrap: normal; width: 200px; } Running your app Start your react app with or depending on your preferred package manager. yarn start npm start Testing your app Thanks for staying tuned! Here is the link to . the source code Also published . here