rubygems.org is now using Elasticsearch
---------------------------------------

With it’s first release on 8 February, 2010, elasticsearch is so old it almost feels nostalgic. Today, elasticsearch is used by thousands of companies for full text search, auto completion, [log processing](https://www.elastic.co/products/logstash) and [analytics](https://www.elastic.co/products/kibana).

Rubygems.org’s search functionality was not up to the mark. One must not have high hopes while using [LIKE operator](https://www.postgresql.org/docs/8.3/static/functions-matching.html) anyway. It was slow, inept and you could only search for [rubygem](https://hackernoon.com/tagged/rubygem) names. The _new search_ is, at the very least, way faster.

![](https://pbs.twimg.com/media/C-WGGj-UAAAlx9O.jpg:large)

I would consider it an achievement for elasticsearch that once I went through [their documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) and watched [a couple of videos](https://www.youtube.com/playlist?list=PLFa6OjVCNd48UAq06jtgiqcRGl03fvGMx), I was able to make meaningful changes to the existing implementation. Also, docs with usage examples are the best kind of docs ❤

### Search multiple fields

By default, queries will be matched against name, description and summary of gems. Each matched result has a relevance score, which is determined by following three factors:

**Term Frequency** \- More often the query term appears in name, description or summary, higher is the relevance score.

**Inverse Document Frequency** - Terms that appear in many documents (words like the, a, gems etc) get lower score than more uncommon terms.

**Field length norm** - Match in longer fields like summary and description get lower score than name. ([Read more](https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html))

![](https://hackernoon.com/hn-images/1*ZyZPyPOtojAeVRF-slGmJA.jpeg)

[Geek And Poke](http://geekandpoke.typepad.com/geekandpoke/2011/01/nosql.html)

It gets better ;) We are using gem downloads to boost search results’ score. It ensures that popular and community approved gems rank higher on the search page.

[Amazon elasticsearch service’s Elasticsearch 5](https://aws.amazon.com/about-aws/whats-new/2017/01/elasticsearch-5-now-available-on-amazon-elasticsearch-service/) support came just in time for us to update our instance before we switch to using it. Elasticsearch 5 has made some [nifty improvements](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_search_changes.html#_scoring_changes) in it’s relevance scoring.

### Filter and advanced search page

In elasticsearch, query string supports a mini-language of its own, which can be used to customize your query. Filter options presented on search page already provide useful defaults to limit your search just to match name, description, summary, updated in last one week or in last one month.

![](https://hackernoon.com/hn-images/1*RkoErYfLqzx19LFPpKthOw.png)

[Search active](https://rubygems.org/search?utf8=%E2%9C%93&query=active) on rubygems.org

You can use [AND](https://rubygems.org/search?utf8=%E2%9C%93&query=ruby+AND+rails) and [OR](https://rubygems.org/search?utf8=%E2%9C%93&query=ruby+OR+rails) operator to combine your query terms. Query string also supports [wildcards](https://rubygems.org/search?utf8=%E2%9C%93&query=active*), [fuzziness](https://rubygems.org/search?utf8=%E2%9C%93&query=activo~), [ranges](https://rubygems.org/search?utf8=%E2%9C%93&query=active+AND+downloads%3A%5B20000+TO+100000%5D), [boosting](https://rubygems.org/search?utf8=%E2%9C%93&query=rails%5E3+db) and [much more](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/query-dsl-query-string-query.html#query-string-syntax). Check out our [advanced search page](https://rubygems.org/search/advanced) when you are in mood for a little experimentation.

### Suggestions and analyzers

Have you been feeling too lazy to type all the characters of your query correctly? Fear not! Elasticsearch suggestions have your back. When no matching results are found, the search page will suggest queries you probably wanted to type.

![](https://hackernoon.com/hn-images/1*sH1_62wdpICIO_AA4CnEcw.png)

[Search raols](https://rubygems.org/search?utf8=%E2%9C%93&query=raols) on rubygems.org

Elasticsearch’s [prepackaged analyzers](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html) have sensible defaults and can be used directly. Analyzers ensure that when you search for [rails async, async rails or async-rail](https://github.com/rubygems/rubygems.org/issues/1412), you get the same result back. We use _pattern_ analyzer over name and _english_ analyzer over summary and description.

### Road behind

![](https://hackernoon.com/hn-images/1*VBtwDGtGfeTOIlOqVJ4bJg.jpeg)

I am not exactly proud of [the code we ended up with](https://github.com/rubygems/rubygems.org/blob/master/app/models/concerns/rubygem_searchable.rb). In name of desperate measure for desperate situation, I wrote [a method](https://github.com/rubygems/rubygems.org/blob/master/app/models/concerns/rubygem_searchable.rb#L52-L62) which returns two values ಠ\_ಠ

There were moments when I either fumbled myself into the correct solution or got [crucial help](https://github.com/elastic/elasticsearch-rails/issues/697) after being stuck. [Learning](https://hackernoon.com/tagged/learning) something new on your own comes with its opportunities and struggles.

Elasticsearch [explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-explain.html) came in handy while tracking scoring issues. The [rails application templates](https://github.com/elastic/elasticsearch-rails/blob/master/elasticsearch-rails/README.md#rails-application-templates) on elasticsearch-rails is a great resource for reference. The elasticsearch-model gem also has [some nice examples](https://github.com/elastic/elasticsearch-rails/tree/master/elasticsearch-model/examples).

### Road ahead

A considerable amount of work is yet to be done. We are yet to switch rubygem.org search api to elasticsearch. We already have [a PR](https://github.com/rubygems/rubygems.org/pull/1315) for adding autocomplete to the search field, however I am a bit skeptical about use of [typehead.js](https://twitter.github.io/typeahead.js/). I am not sure if we need a _suggestion engine_ when _something usable_ can be written with much less code.

Feel free to hit up [our issue board](https://github.com/rubygems/rubygems.org/issues) if you encounter any issue or have suggestions for improvement.

Amazon

Twitter

YouTube

Too Long; Didn't Read

Cassandra Forward a free Cassandra-focused community event

You know, for search..

Too Long; Didn't Read

Companies Mentioned

@sonalkr132

RELATED STORIES