How to Implement Full-text Search in a Laravel Project

Nowadays, implementing full-text search is one of the most important parts of application development. In applications that rely heavily on search functionality, such as music services or online stores, the configuration and customization of answer ranking is vital. In order to better understand why you need to implement such a solution, let's consider the problems in its absence

Enhancing Search Functionality

Often developers are tasked with improving search in an application, and in many cases, quite a lot of time is spent putting together a clear set of refinements for the project.

Common tasks to improve search might include the following:

• Speeding up search performance
• Add error tolerance
• Add the ability to search beyond the item name
• Add the ability to search by words not necessarily in the same order
• Add the ability to search other forms of words from a query
• Add the ability to manage search rules for the same word with different values

Evaluation Criteria

The main criteria for evaluating search performance can be defined as the speed of response, completeness and accuracy. While response rates can be easily measured and evaluated, the other two are a bit more difficult to calculate.

Search completeness is the result of dividing the number of items returned by the number of matching items. When solving a problem of the first kind, with the exclusion of matching items, the problem of an incorrect query with an error or typo is often solved as well.

The precision of the search is the result of dividing the number of matching items in the answer by the number of items in the answer. Those items that made it into the answer, but should not have, are called errors of the second kind.

Accordingly, we need the fastest answer, with the highest percentages of completeness and search accuracy. The balance between these elements is unique to each project. Let's try to consider possible solutions to this problem without implementing full-text search.

Attempting a solution via SQL LIKE

The simplest solution may be the use of LIKE in SQL. If you try to implement this search on several fields, the query can look like this:

SELECT p.id,
  p.title,
  p.image,
  u.name,
  p.created_at
FROM posts p
JOIN users u ON p.author_id = u.id
WHERE p.title LIKE '%query%'
  OR p.text LIKE '%query%'
  OR u.name LIKE '%query%'
  OR u.email LIKE '%query%';

However, this is far from the most optimal variant and if the records in the posts table are more than 1000, then even considering that the indexes are set, the queries will still be processed on the verge of timeouts.

Furthermore, this type of query is appropriate solely for addressing a specific task, where all fields are utilized in the search. However, other tasks, notably rearranging the word order within the query, prioritizing elements, and enhancing the query's search speed, remain largely unresolved.

Full-text search in PostgreSQL/MySQL

The second solution with the existing set of tools in the application can be to use the full-text search features of PostgreSQL and MySQL. In this case, the query will be of the following form: SELECT ... MATCH (field) AGAINST ('word').

The decision to use SQL gives us 100% accuracy and completeness of search but also has a large number of disadvantages. Among them are the need for constant customization of the database and limitations on the type of fields and lack of ranking, as well as the final cost of such a solution.

Laravel Scout

To solve all of the above problems there are special services that quickly produce full-text search and solve problems with tokenization and morphological analysis. Laravel Scout allows you to work with many of these services and provides a unified interface for working with search, as well as linking entity search with Laravel models.

Below is a list of popular tools that work with full-text search. Bolded are those tools that have a driver to work with Laravel Scout. Underlined are those that have such a driver out of the box.

• ElasticSearch

• Apache Solr

• Sphinx

• Algolia

• MultiSearch

• MeiliSearch

  
  

The principle of working with full-text in Laravel does not differ much depending on the driver. I chose Algolia to demonstrate how it works. Laravel Scout makes it easy to add data to the Algolia index and just as easy to retrieve it from there by searching many fields. Of course, when choosing any vendor, your project will acquire a dependency that needs to be kept in mind. Laravel Scout will help make the dependency less, thanks to its interfaces.

Customization Features

To customize Algolia, we need to download the algolia/algoliasearch-client-php package. and set the id and secret from your application to Algolia. This is all we need for Algolia and our application to work and communicate properly. It's also worth noting that for the production solution, we need to add full queue support, as under the hood, when updating data in the selected service asynchronously, that can speed up the search performance a lot. Let's consider our further actions with full-text search on the example of Algolia.

Working with full-text search on Algolia

Service Algolia is partially free, but its minimum free package is probably not enough for any popular site. The important thing to emphasize is that if you choose other options that will run on your servers, such as the popular ElasticSearch, the server, and support costs can easily overlap the cost of using Algolia.

After creating the application and adding credentials to the .env file and customizing the cache, we need to connect the Searchable trait to the models we need. After that the standard customization is complete, however, in some cases, we need to add a few customizations.

Additional settings

For example, in the model, we can specify which fields we will stack in Algolia using the toSearchableArray() method

public function toSearchableArray(): array
{
   return [
       'id' => $this->id,
       'title' => $this->title,
       'author_id' => $this->author_id
   ];
}

Also, if the primary key of our model is not id, we can change it and add its name, for example, this way:

public function getScoutKey(): string
{
   return $this->title;
}


public function getScoutKeyName(): string
{
   return 'title';
}

Laravel Scout also has the option to set the search engine to search separately for each model using the searchableUsing() method.

Indexing records

Most often, full-text search is implemented in a project after a large number of records have appeared in the project. In order to index these records in Algolia we need to use the command php artisan scout:import "App\Models\Post".

You can also add special conditions or related models when indexing using the makeAllSearchableUsing() method if needed. After indexing, you can see the indexed records in Algolia in an index formed by the same principles as the table name in migrations i.e for Post, there will be posts.

If you see the records here, the indexing was successful.

Further adding and updating of indexed records will happen automatically as Laravel Scout listens to events in the model.

It is also possible to add and modify indexes via a method to the Builder or to the Collection that contains the Model objects

Post::where('id', '>', 100)->searchable();

Or

$user->posts()->searchable();
//OR
$posts = Post::where('id', '>', 100)->get();
$posts->searchable();

For the last case, we can specify additional links using a separate method makeSearchableUsing()

If we need to limit the records that should be included in the index when using save() or update(), we can apply the shouldBeSearchable() method

public function shouldBeSearchable(): bool
{
   return $this->status === 'published';
}

Deletion

To remove records from the index, simply use the unsearchable() method for the same Model, Builder, Models Collection, and relation entities. Also in Laravel Scout, you have the option to enable SoftDelete, so that deletion from the database is not so critical.

Search

Finally, we get to the most important thing - checking that our full-text search works. In order to test this, we can use the search() method.

Post::search('repudiandae')->get();

Also, our model can be in several indexes at the same time for different checks. In this case, we specify the index in the within() method.

Additionally, Scout search has the ability to use pagination through the paginate() method, as well as to use simple checks in where(). Thus a query can look like this:

Post::search('repudiandae')
   ->within('another_index_with_posts')
   ->where('author_id',  $authorId)
   ->paginate(15);

Customizing search

The search method searches all fields by default. To change this, you need to specify the list of fields in the index that should be checked.

Algolia already has error tolerance by default, but this can also be changed in the Typo-tolerance tab. As you may have noticed in the previous screenshot, there are a large number of full-text search settings.

Often, to properly handle customization, Algolia requires a clear set of rules that determine which element to reflect above. You can also configure stop words or set up context for certain words and queries. Once full-text is implemented, there is supposed to be a lot of ongoing work to tune it to continually increase the completeness and accuracy of the search.

To understand why the tool selected a particular element and why one is higher and the other lower, you can use the raw() method instead of get().

Conclusion

As we have seen today, implementing full-text search is an important task for a large number of projects, but the initial setup may not take much time. Implementing search right away will give a big boost in search performance and search efficiency. The important issues in this matter are the choice of tool and the subsequent customization iterations.