It’s remarkable how so many things are made better with great searches. Google made it easy for normal folks to find whatever they needed online, no matter how obscure. IDEA’s fuzzy matching and symbol search helped programmers forget the directory structure of their code bases. AirTag added an advanced spatial location to my cat. A well-crafted discovery feature can help add that “wow” factor that iconic, habit-forming products have. In Search of Meaning is a search method for surfacing highly relevant results based on the of the query, context, and content. It goes beyond simple keyword indexing or filtering. Semantic Search meaning It allows users to find things more naturally and with better support nuance than highly sophisticated but rigid traditional relevancy methods. In practice, it feels like the difference between asking a real person or talking to a machine. Tech companies from all over the world are racing to add these capabilities to their existing products. Instacart on how they added semantic deduplication to their search experience. published an extensive article Examples of companies implementing some form of semantic search include eBay, Shopee, Ikea, Walmart, and many others. The reason for this rush towards semantic search is simple: more relevant results = happier customers = more money. Discovery, relevancy, and trustworthiness are some of the hardest problems to solve in e-commerce, and an entire ecosystem exists to help companies solve them. Vectors to the Rescue There is an emerging group of highly capable semantic search SaaS offerings. A prime example is Algolia’s NeuralSearch - if you want a top-notch, batteries-included system that will take care of most of the complications of implementing search right, this is a great place to look. Sadly, you are going to pay - a lot. This might be OK for a low to medium-traffic site or a POC, but do your math before you fully commit to them. Don’t worry though: you can still create an awesome semantic search experience even if you have a more down-to-earth budget. It will just require a bit of doing. Many companies working on semantic search today are using - a way of representing meaning as vectors. document embeddings Since semantic search alone may not be able to provide enough relevant hits, traditional full-text indexing is used as a backup method. A feedback loop is added to track user interactions, and use them to provide super relevant results through result re-ranking. This is what the architecture looks like: This system has three key processes: indexing, querying, and tracking. is done by converting a document’s content to an embeddings vector through a text-to-vector encoder (ex: OpenAI’s API) and inserting it into a Vector Database (ex; Qdrant, Milvus, Pinecone, etc.). Indexing Embeddings Documents are also indexed in a traditional full-text search engine (ex: Elasticsearch). This combination is usually referred to as “hybrid search.” relies on encoding incoming queries into vectors (preferably using the same encoder as the previous step) and querying the vector database using them. These results are then combined with traditional full-text results and re-ranked for relevancy. Querying Search re-ranking is usually a complex problem, and often relies on a mix of machine learning and heuristics. involves capturing important user interactions - ex: clicking on results, liking items, etc. - and using these events to update the machine learning models involved in re-ranking. Tracking This provides a feedback loop that uses user input to continuously improve relevancy. Snowplow is an example of a capable tracking system. What’s Next? If you have the budget for a SaaS solution, then congratulations: you are well on your way to impressing your users with a spanking new search function. If, like most of us, you are not made of money, then it’s time to roll up your sleeves. Implementation can be a daunting challenge. If you need any help, I to help you get started. In either case, you should seriously consider whether your users could benefit from semantic search. wrote about the subject It’s a hard problem to solve, but the upside is definitely there and users are getting more used to this raised bar every day. Happy searching!

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Searching for Meaning: What Real World Semantic Search Looks Like

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

How to Win as an Associate Software Engineer

A Bibliography Generator From Browser Bookmarks: How It Works

A Guide to Understanding Indexes in MySQL

Analyzer in Elasticsearch: An Introduction

[Explained] How Does A Search Engine Work?

How to Erase Expired Docs Automatically with MongoDB (TTL index)

How to Win as an Associate Software Engineer

A Bibliography Generator From Browser Bookmarks: How It Works

A Guide to Understanding Indexes in MySQL

Analyzer in Elasticsearch: An Introduction

[Explained] How Does A Search Engine Work?

How to Erase Expired Docs Automatically with MongoDB (TTL index)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps