What it Takes to Build a Really Great Search Engine

Building a successful search takes more than technical smarts, but if done right is one of the most rewarding products you could work on.

Everything is a search problem. From finding the keys in our pocket (thanks, AirTag) to looking up a long-lost lover on LinkedIn — seeking things and answers is a problem that predates civilization.

When historians look back at our lives, they will proclaim how this was the time when humans industrialized search on a scale that touched every facet of our lives. Our era marks the transition from search being used for survival to one of expanding our knowledge.

It’s hard not to see the impact.

An average person conducts three and four searches each day; more times than I can remember to drink water.

Google.com is the most dominant website on the planet, logging approximately 5.6 billion searches daily.

We have been looking for answers to life’s questions for centuries, but it’s never been so easy, expansive, and quick to do so. To cope with the newfound convenience, we have honed skills for sifting the informational zeitgeist and learning to pose questions in ways a machine can understand.

“You have people walking around with all the knowledge of humanity on their phone, but they have no idea how to integrate it. We don’t train people in thinking or reasoning.” David Epstein, Range

Servers, algorithms, and programming languages aid us in this struggle, a compass for mapping the optimal route from query to knowledge.

For those of us building such search tools, our problem is knowing that when our users search for anything, there are few guarantees we will provide relevant answers. So, how best to approach this?

The secret to a great search

Being an excellent search product team is the same as being a great team for anything else. The principles are almost cliche: Understand and help your users solve a valuable problem and work backward from there.

When my team and I built our first search engine on Mendeley (Reference management software for researchers), the key to our unexpected success was understanding that search was not just about helping academics find research papers.

What was important was understanding that finding relevant results for our users was key to a better onboarding experience, leading to improved user retention and product growth.

Thanks to focusing on longer-term product impact, we didn’t get distracted by arbitrary goals like optimizing result relevancy. Instead, we concentrated more on the holistic user experience and prioritized features that built trust, such as explaining result relevancy or assisting users with choosing better keywords.

Had we doubled down on search vanity metrics like response speed or % of clicks in the first 10 results, I doubt we would have achieved our double-digit growth and engagement.

I’ve noticed that many who enter the field of search have technical backgrounds and t’s not hard to understand why. Search products are inherently more technical, speculative and experimental than many other user-facing features.

Having the skills to navigate the technical complexities of search, so you understand its limitations and capabilities is a significant advantage when planning and working with engineers. However, people tend to over-index on this knowledge, which blinds us to opportunities.

For example, when building our search, the goal was to help our users become more productive. Doing a good job, we reasoned would increase user engagement, retention and satisfaction. As a result, everything we chose to learn, research, build and measure around search was based on those goals, not vanity relevancy metrics like DCG (Discounted cumulative gain — a fancy way of measuring the number of clicks on the first couple of results).

Our role is to connect a love of the problem with the available technology.

We need to be technical enough to understand the vocabulary of engineers, but not so much as to place restrictions on our goals based on any present technical limitations. To combat this, I would instead recommend planning around any given technology trend rather than its current absolute, e.g. Graph databases and general language models. Aim to build a product roadmap that anticipates what is likely to exist in the medium-term future, and you will always be prosperous.

The other thing to understand is that everything in search is speculative. Users provide many perspectives, which ensure the quality of results will be interpreted very differently. Rarely are there single correct answers for everyone (a bit like real life). Our users could provide different satisfaction levels for the same results for the same query on the same day.

Finding peace with the ambiguity of search work means discovering ways to reduce uncertainty. The best way to achieve this is by focusing on skills that facilitate robust working methods and ensure strong teamwork.

In other words, practicing the “art” of product management in search more valuable than the “science”: communicating, empathy, leading without authority, having difficult conversations, storytelling, making decisions when you don’t have all the information, dealing with ambiguity, inspiring others, and connecting deeply with users.

I can’t emphasize enough not to fall into traps like ensuring response times need to be under 100 ms or results need to be 80% accurate or whatever three-letter acronym metric is currently in vogue: DCG, eDCG, Rank K, MAP. I mean, be aware of them, but shouldn’t let them dictate goals. We need to remain user-focused, not search-focused.

In many ways, search products are a sum of destructions. Like any good product team, we need to predict a little bit at a time and then verify.

If you find yourself stuck for solutions, the best advice I can offer is to approach complex search problems by building a platform where people can provide their raw components and tailor them to their needs.

For our teams, that means creating clarity and access to relevant information that help people make autonomous decisions. For our users, that means building flexibility into our tools to allow users to identify novel ways of solving their problems.

We don’t always need to understand all the complexities of a user’s problem. Sometimes we just need to build the tools, features, and ways of working that enable us to do things independently.

Search is the Best Product

Interestingly, search changes our perception of the world and our ability to live within it.

It changes to text, changes reading (thank you snippets), how to understand, and even what it means to be literate. Our readers can now easily find connections that an author never intended.

“Search engines have become sense making engines, helping to chart, connect and explore infinite textual maps.” Anne-Laure Le Cunff , founder of Ness Labs

Really what a search is, is a bunch of little problems with a bunch of little solutions.

Building such products brings the responsibility of knowing that we enable anyone to form opinions that justify their actions, beliefs, and lives. These are opinions from which we formulate praise for our friends and contempt for our enemies, our long-term projects, our deepest self-doubts, and our highest hopes. (Paraphrasing philosopher Richard Rorty, 1989)

Our greatest challenge as search people is that users are messy. We have to work knowing that user queries are highly variable and usually not well formulated. The challenge becomes understanding a user’s intent when even the user doesn’t know what they want.

Philosophy rarely rhymes with software engineering, but a big part of building a search is figuring out what your user means.

In the hope of more clicks, I see teams building ever more complex personalization, semantic and segmentation models that aim to filter out irrelevance and boost the familiar.

Morally, I believe this is a mistake.

Recommender or historical click boosting solutions built into search assume that users wish their future results to depend on historical behavior.

It’s an assumption that if executed over a long enough time robs individuals of optionality and limits our ability to express ourselves. One of my favorite philosophers, Isiah Berlin, would call this an attack on our final vocabulary — the set of words we carry about that justifies our actions, our beliefs, our sense of the world, and our place within it). We would call this an “echo chamber”.

If you plan to do such things, always ensure you do so transparently. Medium.com, for example, released a feature that allows you to understand and refine your recommendations. It’s not only good for society but also good for business.

I’ve found that the primary source of a company’s dominance is whether it designs its product and business model to be perfectly aligned with its customers’ interests. “If your model will suffer from perfect transparency with customers, you’re not in an unbeatable position,” says venture capitalist Jesse Beyroutey.

Building a great search is the same as building anything else that’s great: assess the opportunity, then define what needs to be made.

My day looks like many other product managers: I get up in the morning, check metrics, prioritize tasks for sprints, write user stories, refine with my team, and conduct research.

The qualities of a great search team are adaptability, optimism, and humility. If you can fall in love with the problem, then being voracious in taking the time to learn and understand what makes search work becomes second nature.

Here are some resources I strongly recommend:

• Relevant Search, Doug Turnbull and John Berryman (Book)
• What Every Software Engineer Should Know About Search, Max Grigorev
• Measuring Search: Metrics Matter, James Rubinstein
• On Search Leadership, Daniel Tunkelang
• Search Product Management: The Most Misunderstood Role in Search? James Rubinstein
• The Future of Text, Frode Hegland (Book)

What’s so satisfying about working on search is knowing that the tool you build brings people so much value. There are many exciting problems and pieces of technology to play with, plus, you get to work with intelligent people (engineers, analysts, data scientists).

Don’t feel discouraged if you don’t think you have the technical know-how for the job.

A great advantage non-technical product managers bring is a problem-first approach. If nothing else, this is the most crucial skill I find technical PMs forget when working through tricky issues. Though in their defense, one could argue I’m simply being naïve and merely passing the hard work onto others.

The downside of working with search is its ability to induce a sense of relative deprivation upon its makers. The pace of change is fast, with new groundbreaking models or frameworks released monthly. There are many technological distractions, and it’s easy to feel like the world is leaving you behind.

Most of your daily work is less cutting-edge AI and more keeping data up to date, deduplicated, synced, and working at scale. Plus, users will keep coming up with creative ways to break your service.

However, the great thing about search is it’s not a solved problem. Every search solution can look and feel very different.

Doing a great job at search can mean the difference between a person having an informational revelation or missing a lifetime opportunity. How often can you say you were able to work on something that achieved that?

“No learner has ever found that he ran short of subjects to explore. But many people who avoided learning, or abandoned it, find that life is drained dry” Maria Montessori, Italian physician

Also published here.