A Software Engineer based in Indonesia.
Typo is something that often happens and can reduce user’s experience, fortunately, Elasticsearch can handle it easily with Fuzzy Query.
Handling typos is a must if you’re building an advanced autocomplete system with Elasticsearch.
If you want to create a simple one instead, you can read my other articles “Create a Simple Autocomplete With Elasticsearch“.
Fuzzy logic is a mathematics logic in which the truth of variables might be any number between 0 and 1. It is different with a Boolean logic that only has the truth values either 0 or 1.
In Elasticsearch, fuzzy query means the terms in the queries don’t have to be the exact match with the terms in the Inverted Index.
To calculate the distance between query, Elasticsearch uses Levenshtein Distance Algorithm.
Calculating a distance with Levenshtein Distance Algorithm is easy.
You just need to compare the first and second word character by character.
If the character is different, then you can add the distance between the words by one.
Let’s see an example, how to calculate the distance between the common typo word “Gppgle” with the correct word “Google”.
After we calculate the distance between “Gppgle” and “Google” with Levenshtein Distance Algorithm, we can see that the distance is 2.
Handling typo in Elasticsearch with Fuzzy Query is also simple.
Let’s start with making an example of the typo word “Gppgle”.
When we’re using normal Match Query, the Elasticsearch will analyze the query "gppgle" first before searching it into the Elasticsearch.
The only term in the inverted index is “google” and it doesn’t match the term "gppgle". Therefore, the Elasticsearch won't return any result.
Now, let’s try Elasticsearch’s fuzzy in Match Query.
As you can see, with fuzzy, the Elasticsearch returned a response.
We’ve learnt in the before that "gppgle" and "google" have the distance of 2.
In the query, we inserted
instead of a number. Why is it working?
Elasticsearch will determine what fuzziness distance is appropriate if we use
value in the
For 6 characters, the Elasticsearch by default will allow 2 edit distance.
fuzziness is preferable, but you can tune it with an exact number if you want to.
Now, let’s try with an exact number to prove that "gppgle" and "google" have a distance of 2.
When we use
, no result is returned by the Elasticsearch.
, though, the Elasticsearch returned the document "google".
This proves our previous distance calculation of "gppgle" and "google" with Levenshtein Distance Algorithm, in which the result is 2.
In the previous example, we use a fuzzy query as a parameter inside Match Query.
But there is another way to use the fuzzy feature, Fuzzy Query.
Seems to be the same! So, what’s the difference between them?
Fuzzy Query works like just Term Query, the query to Elasticsearch is not analyzed and used raw to search the Inverted Index.
For example, let’s index one more document "Hong Kong" .
Let’s look on what terms the analyzer produces with Elasticsearch’s Analyze API.
As you can see, the
produce two terms, "hong" and "kong".
If you read my other article “Elasticsearch: Text vs. Keyword”, you’d know that if we use a term query to search "Hong Kong" then we won't get any result.
This is because there is no term that has less than 2 edit distance with "Hong Kong" in the Elasticsearch.
Now, Let’s try Fuzzy Query with "Hpng" .
Term "Hpng" in the query and the term "hong" in the Elasticsearch have a distance of two.
Remember that the term queried and the term in the inverted index is case-sensitive, the distance “2” comes from the difference between "Hp" and "ho".
Match Query with Fuzziness parameter
Match Query with fuzziness parameter is more preferable than Fuzzy Query. The analyzer in the query will analyze your query before searching it into the Inverted Index.
Let’s try the same query as we did in the Fuzzy Query’s section.
As expected, both queries returned a result!
The first query, "Hpng Kong" is analyzed into "hpng" and "kong". Both terms "hpng" and "kong" exist in the Inverted Index.
"hpng" and "hong" matched with a distance of 1.
While "kong" and "kong" match perfectly.
One thing to note if you plan to use Match Query is that every of the terms in the query will allow fuzziness.
We can try querying with "hggg kggg" which has an edit distance of 4 with "Hong Kong" using
You can tune the Fuzzy Query to match your use case.
In this section, I will write about the parameters that we can change in the query.
Fuzziness is the heart of Fuzzy Query.
The value that we pass to this parameter is the maximum distance allowed.
There are two types of value that we can pass, an integer for exact maximum distance and
value allows the fuzziness in the query to be dynamic.
We can tune 2 parameters in the
value and write it as
. The query will set fuzziness as 0 if the term length is below the low value. If the term length is between the low and high value, the query will set the fuzziness to 1. Last, If the term length is more than the high value, the query will set the fuzziness to 2.
The Elasticsearch will use 3 and 6 as the default if the low and high value is not determined.
Let’s use an example with a document “Fuzzy Query in Elasticsearch allows you to handle typos”.
We can try some queries to prove the mechanism of
we described earlier.
After querying it, these queries produced a result:
The queries don’t:
will allow your query to calculate the transpositions of two adjacent characters (ab -> ba) as 1 distance.
For example, if we set the
to true, we will get a result if we query with "leasticsearcc".
But if we set it as
, there will be no result from the Elasticsearch.
The Elasticsearch defaults the
setting to true.
We can’t set this setting to the Match Query. The Match Query will always calculate
as 1 distance.
will determine the maximum result you get from your query.
If you set the
to 1 and there is 2 document in the Elasticsearch that are appropriate to your query, the Elasticsearch will only return 1.
applies to shard level. So if you have many shards in the Elasticsearch, even if you set the
to 1, the query might return more results.
The default value for
is the number of prefix characters that is not considered in fuzzy query.
For example, if we set the
to 1, we won't get any result if we query "llasticsearch".
setting defaults to
You can change
parameter if you want to change the scoring of the results.
You can find more information about the rewrite parameter in the Elasticsearch documentation.
Handling a typo in Elasticsearch is very easy and can improve the user’s experience.
The simplest way to handle a typo is to just add
in your Match Query.
If you want to tune the Query, there are some parameters that you can change with the "fuzziness" being the most important.
Thank you for reading until the end!
Previously published at https://codecurated.com/blog/how-to-handle-typos-in-elasticsearch-using-fuzzy-query/
Create your free account to unlock your custom reading experience.