Create a Full Autocomplete Search Application with Elasticsearch, Kibana, NestJS and React

Elasticsearch is a distributed, RESTful search and analytics engine that can handle an expanding range of use cases. It offers a full-text search engine with a multitenant capability, an HTTP web interface, and schema-free JSON documents, all with simple installation.

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

In this article, I will be walking you through how to set up elasticsearch on your PC.

How to setup, install, and configure Elasticsearch for Mac

In order to download Elasticsearch, go to the Elasticsearch Download Page, select the respective operating system and click on the download button as shown below:

After downloading, extract the compressed file using the command below:

$ tar -xzvf elasticsearch-8.4.1-darwin-aarch64.tar.gz

Configuring Elasticsearch

After extracting elasticsearch, depending on your use case, you may want to configure elasticsearch to suit you. You can take a look at the config file using the command below:

$ vi elasticsearch-8.4.1/config/elasticsearch.yml

Cluster: You can have multiple nodes in your cluster, in order to do this, you need to ensure that the cluster name on your nodes matches.

# -------------------------------- Cluster ----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: air-elastic
#

Node: This is an identifier for the individual nodes in your cluster

# -------------------------------- Node ----------------------------------
#
# Use a descriptive name for your cluster:
#
node.name: airscholar-node
#

For other configurations, check elasticsearch.yml to fine-tune the configs to suit your use case.

Starting up Elasticsearch

In order to start up elasticsearch engine use the command below:

$ bin/elasticsearch

When starting up elasticsearch for the first time, you will be presented with a password that needs to be changed:

Changing your password for the first time

You should change your password after starting up the elasticsearch for the first time, of course, you can do that using this command:

bin/elasticsearch-reset-password -u elastic

NOTE: ENSURE YOU KEEP YOUR ELASTIC PASSWORD SAFE, IT WILL BE REQUIRED IN ORDER TO CONNECT TO ELASTICSEARCH

Testing Elasticsearch

In your browser, goto https://localhost:9200 or the specified hostname and port in your elasticonfig.yml. You will be asked for username (elastic) and password (the password you changed to when starting elasticsearch for the first time)

You should see something like this:

{
  "name" : "airscholar-node",
  "cluster_name" : "air-elastic",
  "cluster_uuid" : "7g3LmMYFRmyxTyR6ZI-b7A",
  "version" : {
    "number" : "8.4.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "2bd229c8e56650b42e40992322a76e7914258f0c",
    "build_date" : "2022-08-26T12:11:43.232597118Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

If you have any challenges setting up elasticsearch, feel free to drop a comment, and I will try to respond. You can also check the official elasticsearch documentation for more detailed information about elasticsearch.

Kibana

Kibana is a source-available data visualization dashboard software for Elasticsearch, whose free and open source successor in OpenSearch is OpenSearch Dashboards.

How does Kibana help us in this case you may ask?

Kibana is a great visualization tool that helps us with our elasticsearch queries and manipulation. It is part of B.E.L.K (Beats, Elasticsearch, Logstash, and Kibana) stack. In case you don't know much about it, follow along and you will be just fine! (A separate article on Kibana can be written as an addendum to this series if you want, let me know in the comment section!)

Setting up and Configuring Kibana

In order to download Kibana, go to Kibana Download Page, and click on the download button.

After downloading, extract the zip file using this command:

$ tar -xzvf kibana-8.4.1-darwin-aarch64.tar.gz
$ cd kibana-8.4.1

There are two ways to configure and connect Kibana to elasticsearch, one way is to manually input the required keys into kibana.yml (in config/kibana.yml) or run Kibana and configure it on the UI. I'm choosing the latter 😉!

Run kibana using this command:

bin/kibana

When starting Kibana for the first time, you will be presented with this screen:

There are two ways to tackle this;

Use the elastic enrollment token (usually generated on your terminal when starting elastic for the first time) see below;

Copy and paste the enrollment token above to Kibana and click Configure Elastic. You will be fine.

Manually configure Kibana with elastic using the elastic URL. Click on check address and click Configure elastic.

Enter kibana_system password generated. Or click forgot password (if you can not recall the password), and paste the command below to your terminal to reset the password.

bin/elasticsearch-reset-password --username kibana_system

You will be asked to enter a verification code and check your terminal (where you are running kibana to retrieve it)

That's it! You are all set up.

Enter your elastic username and password to log in to kibana and you are in!

Querying elastic using Kibana

Let's run a sample query to test our connection, shall we?

Click on the hamburger (three lines on the top left corner of your screen)
Scroll down to the bottom of the page
Click on Dev tools

You should see something like this:

On the left pane is where queries are written, and the right pane is where the outputs are presented.

Ensure you click on any part of the query and select the play button.

The output of the query will be presented on the right pane.

Loading data into elasticsearch

To enable us to write our code effectively, we need data loaded into our elasticsearch. We will be using a sample dataset from Kaggle (Download it here).

Follow the FOUR steps below to load it up into elasticsearch:

Open up Kibana (http://localhost:5601)
Under Get started by adding integrations Click on Upload a file:
Click on import and enter the name of the index you want to put the data in.
Click on import (final page)

If you made it to this point, you have successfully imported data into elasticsearch.

Querying for sample

Goto DevTools (Hamburger on the top left corner of the screen > Scroll down to Management > DevTools)

Run the query below (select it and click on the play button)

GET tmdb_movies/_search

If you see this, we are good to go!

Now, Let's dive into coding, shall we 😊?

NestJS

NestJS is a progressive Node. js framework that helps build server-side applications. Nest extends Node. js frameworks like Express or Fastify adding modular organization and a wide range of other libraries to take care of repetitive tasks. It's open-source, uses TypeScript, and is a very versatile Node.

Creating a NestJs Application

Run the command below to install nestcli and create a new NestJs Application (in the article, the name of the app will be nest-elastic).

$ npm i -g @nestjs/cli
$ nest new nest-elastic

You will be asked to select a package manager, you can select npm, yarn, or pnpm. I will be selecting yarn (you can choose any other one you want 😉). Your project will be set up and we should be ready to code!

Adding elasticsearch to your app

Run the command below to add elasticsearch to your nest-elastic and other dependencies:

yarn add @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config

In your root folder add your .env file with the following contents:

ELASTICSEARCH_NODE=https://localhost:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=elasticPasswordGoesHere
ELASTICSEARCH_MAX_RETRIES=10
ELASTICSEARCH_REQ_TIMEOUT=50000
ELASTICSEARCH_INDEX=tmdb_movies

Let's create a separate module that handles only search using elasticsearch. A simple shortcut is to use the command below (you are welcome to do it manually if you want to):

nest g resource search

Update the search.module.ts to have the content below:

import { Module } from '@nestjs/common';
import { SearchService } from './search.service';
import { SearchController } from './search.controller';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { ElasticsearchModule } from '@nestjs/elasticsearch';

@Module({
  imports: [
    ConfigModule,
    ElasticsearchModule.registerAsync({
      imports: [ConfigModule],
      useFactory: async (configService: ConfigService) => ({
        node: configService.get('ELASTICSEARCH_NODE'),
        auth: {
          username: configService.get('ELASTICSEARCH_USERNAME'),
          password: configService.get('ELASTICSEARCH_PASSWORD'),
        },
        maxRetries: configService.get('ELASTICSEARCH_MAX_RETRIES'),
        requestTimeout: configService.get('ELASTICSEARCH_REQ_TIMEOUT'),
      }),
      inject: [ConfigService],
    }),
  ],
  controllers: [SearchController],
  providers: [SearchService],
  exports: [SearchService],
})
export class SearchModule {}

Update search.service.ts with the content below:

import { ConfigService } from '@nestjs/config';
import { Injectable } from '@nestjs/common';
import { ElasticsearchService } from '@nestjs/elasticsearch';

type dataResponse = {
  UnitPrice: number;
  Description: string;
  Quantity: number;
  Country: string;
  InvoiceNo: string;
  InvoiceDate: Date;
  CustomerID: number;
  StockCode: string;
};

@Injectable()
export class SearchService {
  constructor(
    private readonly esService: ElasticsearchService,
    private readonly configService: ConfigService,
  ) {}

  async search(search: {key: string}) {
    let results = new Set();
    const response = await this.esService.search({
      index: this.configService.get('ELASTICSEARCH_INDEX'),
      body: {
        size: 50,
        query: {
          match_phrase: search
        },
      },
    });
    const hits = response.hits.hits;
    hits.map((item) => {
      results.add(item._source as dataResponse);
    });

    return { results: Array.from(results), total: response.hits.total };
  }
}

Now, let's add movies modules:

nest g resource movies

Update movies.controller.ts with the content below:

import { SearchService } from './../search/search.service';
import { Body, Controller, Post } from '@nestjs/common';

@Controller('movies')
export class MoviesController {
  constructor(private readonly searchService: SearchService) {}

  @Post('search')
  async search(@Body() body) {
    return await this.searchService.search(body.data);
  }
}

Then movies.module.ts

import { SearchModule } from './../search/search.module';
import { Module } from '@nestjs/common';
import { MoviesService } from './movies.service';
import { MoviesController } from './movies.controller';

@Module({
  imports: [SearchModule],
  controllers: [MoviesController],
  providers: [MoviesService],
})
export class MoviesModule {}

Finally, update app.module.ts

import { MoviesModule } from './movies/movies.module';
import { ConfigModule } from '@nestjs/config';
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { SearchModule } from './search/search.module';

@Module({
  imports: [MoviesModule, ConfigModule.forRoot(), SearchModule],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

Your package.json should look like this:

{
  "name": "nest-elastic",
  "version": "0.0.1",
  "description": "",
  "author": "Yusuf Ganiyu",
  "private": true,
  "license": "UNLICENSED",
  "scripts": {
    "prebuild": "rimraf dist",
    "build": "nest build",
    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
    "start": "nest start",
    "start:dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/main",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "test": "jest",
    "test:watch": "jest --watch",
    "test:cov": "jest --coverage",
    "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
    "test:e2e": "jest --config ./test/jest-e2e.json"
  },
  "dependencies": {
    "@elastic/elasticsearch": "^8.4.0",
    "@nestjs/common": "^9.0.0",
    "@nestjs/config": "^2.2.0",
    "@nestjs/core": "^9.0.0",
    "@nestjs/elasticsearch": "^9.0.0",
    "@nestjs/mapped-types": "*",
    "@nestjs/platform-express": "^9.0.0",
    "reflect-metadata": "^0.1.13",
    "rimraf": "^3.0.2",
    "rxjs": "^7.2.0"
  },
  "devDependencies": {
    "@nestjs/cli": "^9.0.0",
    "@nestjs/schematics": "^9.0.0",
    "@nestjs/testing": "^9.0.0",
    "@types/express": "^4.17.13",
    "@types/jest": "28.1.4",
    "@types/node": "^16.0.0",
    "@types/supertest": "^2.0.11",
    "@typescript-eslint/eslint-plugin": "^5.0.0",
    "@typescript-eslint/parser": "^5.0.0",
    "eslint": "^8.0.1",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-prettier": "^4.0.0",
    "jest": "28.1.2",
    "prettier": "^2.3.2",
    "source-map-support": "^0.5.20",
    "supertest": "^6.1.3",
    "ts-jest": "28.0.5",
    "ts-loader": "^9.2.3",
    "ts-node": "^10.0.0",
    "tsconfig-paths": "4.0.0",
    "typescript": "^4.3.5"
  },
  "jest": {
    "moduleFileExtensions": [
      "js",
      "json",
      "ts"
    ],
    "rootDir": "src",
    "testRegex": ".*\\.spec\\.ts$",
    "transform": {
      "^.+\\.(t|j)s$": "ts-jest"
    },
    "collectCoverageFrom": [
      "**/*.(t|j)s"
    ],
    "coverageDirectory": "../coverage",
    "testEnvironment": "node"
  }
}

Running the app

You can fire up the app in the dev environment using yarn start:dev

Testing

You can access the full source code here.

Setting up a react project

You can set up a simple react app using this command (or check out this detailed react documentation):

$ npx create-react-app nest-elastic-frontend

Once your app is set up, open it in your favorite IDE, mine is VSCode.

We will need to install axios as a dependency. If you prefer npm package manager, you will have to run npm install axios but if you preferred yarn, yarn add axios.

We need to update three files

public/index.html

You need to add bootstrap CDN. (PS: I have removed the comments to reduce the length of the file).

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta name="description" content="Web site created using create-react-app" />
    <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <link
      href="//maxcdn.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css"
      rel="stylesheet"
      id="bootstrap-css"
    />
    <title>React App</title>
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    <script src="//maxcdn.bootstrapcdn.com/bootstrap/4.1.1/js/bootstrap.min.js"></script>
    <script src="//cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
  </body>
</html>

src/App.js

import './App.css';
import axios from 'axios';
import { useState } from 'react';
const App = () => {
  const [searchResponse, setSearchResponse] = useState([]);
  const [totalValue, setTotalValue] = useState();

  const handleChange = async e => {
    const { data } = await axios.post('http://localhost:8000/movies/search', {
      data: {
        title: e.target.value,
      },
    });

    setSearchResponse(data.results);
    setTotalValue(data.total.value);
  };
  return (
    <div className='App'>
      <div className='container search-table'>
        <div className='search-box'>
          <div className='row'>
            <div className='col-md-3'>
              <h5>Search All Fields</h5>
            </div>
            <div className='col-md-9'>
              <input
                type='text'
                id='myInput'
                onChange={handleChange}
                className='form-control'
                placeholder='Search IMDB movies'></input>
            </div>
          </div>
        </div>
        <div className='search-list'>
          <h3>
            {totalValue ?? 0} {totalValue > 1 ? 'Records' : 'Record'} Found
          </h3>
          <table className='table' id='myTable'>
            <thead>
              <tr>
                <th>Title</th>
                <th>Overview</th>
                <th>Revenue:Budget ($)</th>
              </tr>
            </thead>
            <tbody>
              {searchResponse.map((res, idx) => (
                <tr key={idx}>
                  <td className='title'>{res.title}</td>
                  <td>
                    <p>{res.overview}</p>
                    <sub>"{res.tagline}"</sub>
                  </td>
                  <td>
                    <p>
                      <sub>
                        {res.revenue.toLocaleString()}:{res.budget.toLocaleString()}
                      </sub>
                    </p>
                  </td>
                </tr>
              ))}
            </tbody>
          </table>
        </div>
      </div>
    </div>
  );
};

export default App;

Lastly, src/index.css

body {
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu',
    'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

code {
  font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace;
}

.search-table {
  padding: 10%;
  margin-top: -6%;
}
.search-box {
  background: #c1c1c1;
  border: 1px solid #ababab;
  padding: 3%;
}
.search-box input:focus {
  box-shadow: none;
  border: 2px solid #eeeeee;
}
.search-list {
  background: #fff;
  border: 1px solid #ababab;
  border-top: none;
}
.search-list h3 {
  background: #eee;
  padding: 3%;
  margin-bottom: 0%;
}

.title {
  word-wrap: normal;
  width: 200px;
}

Running your app

Start your react app with yarn start or npm start depending on your preferred package manager.

Testing your app

Thanks for staying tuned!

Here is the link to the source code.

Also published here.