3,817 reads

We Built A Search Engine With MeiliSearch and JavaScript: Here's How You Can Too

by Michiel MuldersNovember 9th, 2020

Too Long; Didn't Read

MeiliSearch is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box. Throughout this tutorial, we’ll be using a cool Nobel prize winners data set to show you a couple of examples. To keep things simple, we aren’t using a master key to protect all API endpoints for your Meilisearch instance.

Companies Mentioned

featured image - We Built A Search Engine With MeiliSearch and JavaScript: Here's How You Can Too

Struggle to find the chemistry Nobel prize winner for 1979? Try out MeiliSearch's super-fast-search JavaScript tutorial to find Nobel prize winners in a matter of milliseconds. 🌬️

MeiliSearch JS is a client for MeiliSearch written in JavaScript. MeiliSearch is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box.

A working example can be found on

and in the meilisearch-vue directory.

The goal of this project is to teach you how to:

Create a new index
Modify settings for your index
Add data to your index
Ignore stop words
Enable facet filters for faster data filtering

We’ll be using both the API directly as the JavaScript wrapper to show you how both work. Throughout this tutorial, we’ll be using a cool Nobel prize winners data set to show you a couple of examples.

First, let’s take a look at the requirements.

Requirements

Here are the requirements to be able to follow this tutorial:

Node.js installation
Running an instance of MeiliSearch (via Docker or a DigitalOcean droplet) Don’t want to set up your own MeiliSearch instance, try out our MeiliSearch sandbox to create a temporary index for 72h.
cURL for sending requests from the terminal (Postman works as well)
MeiliSearch-js wrapper and dependencies (see installation guide).

Let's get started with the creation of your first index.

Project setup and MeiliSearch-js installation

To follow along, we need to set up our JavaScript project and install MeiliSearch-js. Create a new folder and run the following command in your terminal.

npm init -y

This will prepare your project setup. Next, we can add the MeiliSearch-js dependency.

npm install meilisearch

Lastly, let’s create a file called index.js in your project. We’ll use this file to add our JavaScript code.

touch index.js

Good, let’s get started!

Step 1: Create your first index

First of all, let’s start with the preparation. We assume here that you have a running instance of MeiliSearch and can access this via your localhost or a public IP address.

Important: To keep things simple, we aren’t using a master key. A master key allows you to protect all API endpoints for your Meilisearch instance. We highly recommend setting a master key when using Meilisearch in production or hosting it via a publicly accessible IP address - e.g. DigitalOcean droplet.

To verify if you can reach your MeiliSearch instance, try querying for the available indexes. If you haven’t created any indexes, you should see an empty array as a result. Below you find the cURL command you can execute from your terminal.

curl http://127.0.0.1:7700/indexes

Now, let’s edit our

index.js

file to create a connection object. We’ll use this connection object to create our first index. First, we need to import the MeiliSearch dependency. Furthermore, the

host

property accepts the IP address of your MeiliSearch instance.

const MeiliSearch = require('meilisearch')
 
const main = async () => {
    const client = new MeiliSearch({
        host: 'http://127.0.0.1:7700'
    })
 
    const indexes = await client.listIndexes()
    console.log(indexes)
}
 
main()

Note that we’ve added some extra code that uses the

client

object to query for all indexes and then prints the result.

To execute the file, you can run it with the

node

command.

node index.js

Finally, let’s create our first index. As we are working with Nobel prizes, let’s name the index

prizes

. We can use the

createIndex

function to create a new index. To verify the successful creation of our index, let’s query again for all indexes to see the newly created index.

const MeiliSearch = require('meilisearch')
 
const main = async () => {
    const client = new MeiliSearch({
        host: 'http://127.0.0.1:7700'
    })
 
    const indexes = await client.listIndexes()
    console.log(indexes)
 
    await client.createIndex('prizes')
 
    const updatedIndexes = await client.listIndexes()
    console.log(updatedIndexes)
}
 
main()

You should see the following result being printed to your terminal.

[ 
  { 
    name: 'prizes',
    uid: 'prizes',
    createdAt: '2020-10-21T16:08:23.052004747Z',
    updatedAt: '2020-10-21T16:08:23.052024846Z',
    primaryKey: null 
  }
]

Note: The

name

property is optional. By default, Meilisearch sets the

name

property equal to the

uid

property. However, when dealing with a lot of similar indexes, it’s useful to define a human-readable name that differs from the

uid

property.

Also, Meilisearch hasn’t set a primary key yet for the prizes index. When we add data in the next step, the primary key will be automatically picked as our data set contains an id field.

Index created? Good! Let’s explore the Nobel prizes data set.

Step 2: Adding the Nobel prizes dataset

First of all, let’s explore the dataset briefly. The original dataset we used for this example comes from nobelprize.org, however, we’ve modified the dataset slightly to fit our use case.

You can explore the modified dataset here. The structure of the data looks like this.

Each object contains an ID that will serve as the primary key. MeiliSearch will automatically search for a property that contains the word

id

, such as

prizeId

objectId

. If you want to use a dataset that doesn’t contain an

id

, you can still manually set a primary key.

Further, we find properties such as year, category, firstname, surname, motivation, and share.

{
    id: "991",
    year: "2020",
    category: "chemistry",
    firstname: "Emmanuelle",
    surname: "Charpentier",
    motivation: "for the development of a method for genome editing",
    share: "2"
}

Now, let’s download the dataset as a JSON file using cURL. We are using the

-o

property to define an output file for the downloaded contents.

curl -L https://gist.githubusercontent.com/michielmulders/5328ffcabd04bdcaded389a278728304/raw/3c172f0bbc248f99096c2e34653432b983704f07/prizes.json -o prizes.json

Next, we need to add the dataset to the MeiliSearch instance. Let’s upload the dataset to the

prizes

index. Note that the URL changed slightly as we are adding

documents

to the

prizes

index:

indexes/prizes/documents

. Make sure the filename for the

--data

property matches the filename of your prizes JSON file.

curl -i -X POST 'http://127.0.0.1:7700/indexes/prizes/documents' \
  --header 'content-type: application/json' \
  --data @prizes.json

To verify if the data has been uploaded successfully, let’s query for all documents. You should see all Nobel prize objects.

curl http://127.0.0.1:7700/indexes/prizes/documents

Success! Next, let’s use some code to add an extra document to our

prizes

index.

Step 2.1: Add documents using the MeiliSearch-js client

We’ve just added documents using our terminal. It’s time to add an extra document using JS code. Let’s define an array that contains new documents we want to add to our

prizes

index. Note, we first need to retrieve our index so we can use this index object to add documents.

const MeiliSearch = require('meilisearch')
 
const main = async () => {
    const client = new MeiliSearch({
        host: 'http://127.0.0.1:7700'
    })
 
    const index = client.getIndex('prizes')
    const documents = [
        {
            id: '12345',
            year: '2021',
            category: 'chemistry',
            firstname: 'Your',
            surname: 'Name',
            motivation: 'for the development of a new method',
            share: '1'
        }
    ]
 
    let response = await index.addDocuments(documents)
    console.log(response) // => { "updateId": 0 }
}
 
main()

When you add a new document, Meilisearch returns an object containing an

updateId

. Using the update methods you can track the document addition process until it is processed or failed.

In step 3, let’s learn how to search for documents.

Step 3: Search for Nobel prize documents

Searching for documents is pretty simple. Again, we first need to retrieve the index object. Next, we can use the index object to search for a particular query. As an example, we are looking for

chemisytr

to show MeiliSearch’s type-tolerance.

const MeiliSearch = require('meilisearch')
 
const main = async () => {
    const client = new MeiliSearch({
        host: 'http://127.0.0.1:7700'
    })
 
    const index = client.getIndex('prizes')
    const search = await index.search('chemisytr')
    console.log(search)
}
 
main()

This returns a huge list of results. Let’s learn how you can add filters to, for example, limit the number of results. Change the following line to add an object that accepts filters.

 const search = await index.search('chemisytr', { limit: 1})

This returns the following result.

{ 
  hits: [ 
     { 
       id: '991',
       year: '2020',
       category: 'chemistry',
       firstname: 'Emmanuelle',
       surname: 'Charpentier',
       motivation: '"for the development of a method for genome editing"',
       share: '2' 
     } 
  ],
  offset: 0,
  limit: 1,
  nbHits: 111,
  exhaustiveNbHits: false,
  processingTimeMs: 1,
  query: 'chemisytr' 
}

Next, we want to modify the settings for the prizes index to eliminate stop words.

Step 4: Modify index settings to eliminate stop words

Now, let’s take a look at the settings for our prizes index. You can access the settings via the exposed API like so:

http://localhost:7700/indexes/prizes/settings

You’ll see the following result with an empty

stopWords

array.

{
  rankingRules: [
    "typo",
    "words",
    "proximity",
    "attribute",
    "wordsPosition",
    "exactness"
  ],
  distinctAttribute: null,
  searchableAttributes: ["*"],
  displayedAttributes: ["*"],
  stopWords: [ ],
  synonyms: { },
  attributesForFaceting: [ ]
}

We can achieve the same using JavaScript code like so.

 const index = client.getIndex('prizes')
 const settings = await index.getSettings()
 console.log(settings)

Now, let’s add a couple of stop words we want to eliminate. Stop words are frequently occurring words that don’t have any search value.

For example, no products exist that are called

the

. To improve the search speed, we want to avoid searching for such stop words. When the user looks for the search query “a mask”, the MeiliSearch engine will automatically remove the

part and look for the word

mask

In this example, we want to eliminate the following stop words:

First, let’s check how many results we receive when querying for the word

the

 const index = client.getIndex('prizes')
 const results = await index.search('the')
 console.log(results.nbHits)

The above query for

the

returns 494 hits. Now, let’s modify our

index.js

script to eliminate the above stop words.

 const index = client.getIndex('prizes')
 const response = await index.updateSettings({
   stopWords: ['a', 'an', 'the']
 })
 console.log(response)

To verify the effectiveness of our settings change, let’s query again for the word

the

. Now, this query should return

results. Cool right?

Quick tip: You might have forgotten a particular stop word such as

and

. If you send a new

updateSettings

request to your API, this will overwrite the old configuration. Therefore, make sure to send the full list of stop words every time you want to make changes.

Let’s move on!

Faceted search is a feature provided out-of-the-box by MeiliSearch. Faceting allows classifying search results into categories that are called facets. This allows the MeiliSearch engine to search through data much faster using facet filters.

Facet filters are most useful for numbers or enums. For example, Nobel prizes are awarded for a fixed list of categories. This makes up a great facet filter. The same is true for the year property.

Note: You can only use facet filters for strings or arrays of strings. In our example, the

year

property is stored as a string which allows us to filter by the

year

property.

Below you find an example snippet that adds facet filters for the properties

year

and

category

. You can always verify which facet filters have been added by consulting the settings for your index.

 const index = client.getIndex('prizes')
  const response = await index.updateSettings({
    attributesForFaceting: ['category', 'year']
  })

Now, I want to query all Nobel prize winners with the name Paul. This returns 14 results.

 const index = client.getIndex('prizes')
 
 const search1 = await index.search('paul')
 console.log(`Search 1 hits: ${search1.nbHits}`) // 14

Following, I want to filter down the results by the category chemistry. Note that we send an extra data property with the request that allows us to set facet filters. This property accepts an array of strings. Each string contains two words, separated by a colon. The first part is the facet and the second part contains the value. That’s how we get the following facet filter for

chemistry: category:Chemistry

 const index = client.getIndex('prizes')
 
 const search2 = await index.search('paul', { facetFilters: [ 'category:Chemistry' ]})
 console.log(`Search 2 hits: ${search2.nbHits}`) // 5

This query returns five results.

Lastly, I want to add some extra facet filter to filter by both category and year. I want to return Nobel prize winners for the year 1995, 1996, or 1997. Luckily, MeiliSearch allows for combining multiple facet filters. If you want to add multiple values for the same facet filter, you’ll have to use another array inside the facet filters array. Look at the example query below.

 const index = client.getIndex('prizes')
 
 const search3 = await index.search('paul', { facetFilters: [ 'category:Chemistry', ['year:1995', 'year:1996', 'year:1997'] ]})
 console.log(`Search 3 hits: ${search3.nbHits}`) // 2

Ultimately, this returns only two results that match our needs.

{ 
  hits:[ 
    { 
       id: '287',
       year: '1997',
       category: 'chemistry',
       firstname: 'Paul D.',
       surname: 'Boyer',
       motivation:
        '"for their elucidation of the enzymatic mechanism underlying the synthesis of adenosine triphosphate (ATP)"',
       share: '4' 
     },{ 
       id: '281',
       year: '1995',
       category: 'chemistry',
       firstname: 'Paul J.',
       surname: 'Crutzen',
       motivation: '"for their work in atmospheric chemistry, particularly concerning the formation and decomposition of ozone"',
       share: '3' 
     }
  ],
  offset: 0,
  limit: 20,
  nbHits: 2,
  exhaustiveNbHits: false,
  processingTimeMs: 0,
  query: 'paul' 
}

Nice! Lastly, let’s play with ranking rules for our MeiliSearch engine.

Step 6: Play with MeiliSearch ranking rules

In step 3, we’ve shown you how MeiliSearch handles typos by querying for

chemisytr

instead of

chemistry

However, you might have noticed that the settings for your index list many different ranking rules. Ranking rules are what defines relevancy in MeiliSearch. They affect how a result is considered better than another. Ranking rules are sorted top-down by order of importance.

We can change the order of these ranking rules based on your use case to fine tune the speed of your MeiliSearch instance.

It’s even possible to remove certain ranking rules.

{
  rankingRules: [
    "typo",
    "words",
    "proximity",
    "attribute",
    "wordsPosition",
    "exactness"
  ],
  distinctAttribute: null,
  searchableAttributes: ["*"],
  displayedAttributes: ["*"],
  stopWords: ["a", "an", "the"],
  synonyms: { },
  attributesForFaceting: [
    "year",
    "category"
  ]
}

Let’s query for

Paul

again with the above standard ranking rules. We are most interested in results 10 to 14 in the returned list. To get this list, we use the

slice

method for arrays.

    const index = client.getIndex('prizes')
    const search = await index.search('Paul')
    console.log(search.hits.slice(10, 14))

We get the following result. Note that results with id

and

are listed first.

[ 
  { 
    id: '737',
    year: '2001',
    category: 'medicine',
    firstname: 'Sir Paul',
    surname: 'Nurse',
    motivation: '"for their discoveries of key regulators of the cell cycle"',
    share: '3' 
  },
  { 
    id: '637',
    year: '1964',
    category: 'literature',
    firstname: 'Jean-Paul',
    surname: 'Sartre',
    motivation:
     '"for his work which, rich in ideas and filled with the spirit of freedom and the quest for truth, has exerted a far-reaching influence on our age"',
    share: '1'
  },
  { 
    id: '217',
    year: '1954',
    category: 'chemistry',
    firstname: 'Linus',
    surname: 'Pauling',
    motivation: '"for his research into the nature of the chemical bond and its application to the elucidation of the structure of complex substances"',
    share: '1' 
  },
  { id: '50',
    year: '1945',
    category: 'physics',
    firstname: 'Wolfgang',
    surname: 'Pauli',
    motivation: '"for the discovery of the Exclusion Principle, also called the Pauli Principle"',
    share: '1'
  }
]

Now, let’s change the ranking rules where we prefer the

wordsPosition

rule to be the most important rule. From the documentation, we can find the following information about the wordsPosition rule.

“Results are sorted by the position of the query words in the attributes: find documents that contain query terms earlier in their attributes first.”

For our example, we want to prioritize names that start with the word

Paul

over composed names such as

Sir Paul

Jean-Paul

. Let’s change the order first using our code example.

const index = client.getIndex('prizes')
await index.updateSettings({
        rankingRules:
            [
                "wordsPosition",
                "typo", 
                "words", 
                "proximity", 
                "attribute",
                "exactness"
            ]
    })

Now, let’s query for Paul again. Notice how the position for both

Sir Paul

and

Jean-Paul

has changed.

[ { id: '217',
    year: '1954',
    category: 'chemistry',
    firstname: 'Linus',
    surname: 'Pauling',
    motivation:
     '"for his research into the nature of the chemical bond and its application to the elucidation of the structure of complex substances"',
    share: '1' },
  { id: '50',
    year: '1945',
    category: 'physics',
    firstname: 'Wolfgang',
    surname: 'Pauli',
    motivation:
     '"for the discovery of the Exclusion Principle, also called the Pauli Principle"',
    share: '1' },
  { id: '737',
    year: '2001',
    category: 'medicine',
    firstname: 'Sir Paul',
    surname: 'Nurse',
    motivation:
     '"for their discoveries of key regulators of the cell cycle"',
    share: '3' },
  { id: '637',
    year: '1964',
    category: 'literature',
    firstname: 'Jean-Paul',
    surname: 'Sartre',
    motivation:
     '"for his work which, rich in ideas and filled with the spirit of freedom and the quest for truth, has exerted a far-reaching influence on our age"',
    share: '1' } ]

You can find more examples for each ranking rule in the MeiliSearch documentation.

Step 6.1: Define your own ranking rule

Lastly, it’s possible to define your own ranking rule. That’s pretty exciting right? You should know that custom ranking rules only work for numeric values. However, it doesn’t matter that you’ve defined a numeric value as a string such as our

year

property. Now, let’s get funky and remove all ranking rules and add a custom ranking rule for the

year

property.

You can define an ascending or descending sorting rule.

    const index = client.getIndex('prizes')
    await index.updateSettings({
        rankingRules:
            [
                "desc(year)"
            ]
    })

Next, let’s search for

Paul

again. Now, notice that the results are sorted by the

year

property as expected.

[ 
  { 
    id: '995',
    year: '2020',
    category: 'economics',
    firstname: 'Paul',
    surname: 'Milgrom',
    motivation: '"for improvements to auction theory and inventions of new auction formats"',
    share: '2'
   },
   { 
     id: '834',
     year: '2008',
     category: 'economics',
     firstname: 'Paul',
     surname: 'Krugman',
     motivation: '"for his analysis of trade patterns and location of economic activity"',
     share: '1'
   },
   { 
     id: '764',
     year: '2003',
     category: 'medicine',
     firstname: 'Paul C.',
     surname: 'Lauterbur',
     motivation: '"for their discoveries concerning magnetic resonance imaging"',
     share: '2'
   },
   …
]

That’s it!

Conclusion

That’s it for this MeiliSearch and JS tutorial. This tutorial has taught you how to use the MeiliSearch API, create indexes, modify index settings, and define facet filters for more accurate and faster searching.

For more information, make sure to take a look at the documentation and the JS API wrapper on GitHub.

Liked using MeiliSearch, make sure to show us some love by giving MeiliSearch a star on GitHub!

Previously published at https://blog.meilisearch.com/how-to-search-nobel-prize-winners-faster-with-meilisearch-api-for-javascript/