3 Biggest Barriers to Efficient, Relevant AI Powered Search

Consumers expect always-on, frictionless, dynamic, and personalized digital experiences. But as it stands, 99 percent of companies not named Amazon or Netflix have struggled to deliver world-class search. Why? Because humans search in messy, unpredictable ways. People use different words to look for the same things – and this doesn’t even take into account the different languages we speak. Expected search results vary from one person to another based on preferred brands and lifestyles. Expected results also change over time based on seasons and trends. (Think about what we expected when searching for “masks” today compared to a year ago.) Also, results customers expect don’t always align with what businesses want to surface first. Finally, there is a vast difference between a site that delivers “basic textual search” and one that provides “best-in-class search.” Best-in-class search is fast, relevant, personalized, and predictive/prescriptive — and this is the type of search that fuels the best digital experiences. Enter Artificial Intelligence (AI). Businesses can apply AI to solve search-related challenges, thereby enabling brands to deliver meaningful digital experiences. When it comes to applying AI to search, many organizations find the task complex and challenging. Tech leaders consider AI difficult to master; they think it is a technology that is not “standardized” that can yield sometimes unexplainable results. AI is also a technology that requires specific expertise to implement, and a great deal of testing, iterating and fine-tuning. What are the biggest barriers to adopting AI-powered search, and how can we begin to address these roadblocks? First, let’s discuss what we mean when we say “AI-powered search.” Search is inherently complex, with ever-changing user behavior and ever-expanding (yet imperfect) data. And while AI can help simplify the search process and improve the accuracy of search results, AI is not magic, “one-size-fits-all” solution. But we can solve many of the search’s complexities with a step-by-step approach. We want to build a search engine that gets smarter and learns from user behavior. AI should allow companies to deploy tailored digital experiences that are driven by transparency, natural language understanding, and personalization. Ultimately, we want people to consistently find what they’re looking for in their top three search results. Ideally, when someone asks their Alexa or Google Home assistant a question, we want them to get the single best answer. So, what roadblocks stand between companies and their AI-powered search goals? This means there are a lot of different technologies and tools associated with AI, and each is complex to master. There is also an expectation mismatch between what enterprises expect of AI and what AI can actually do. Sometimes, a problem or use case needs to be broken down into a series of smaller, more concrete problems that existing AI technology can solve. Search, for example, has multiple problems: data enrichment/cleanup issues, natural language processing challenges, synonym differences, query understanding disconnects, etc. Currently, no single AI algorithm exists that is able to solve all these problems at once. Without an AI standard, mapping problems to specific technologies can be a challenge. Adopting AI requires a great deal of expertise, testing, and expensive resources. Think of the SQL standard for accessing and working with databases. I think an AI-related equivalent to the SQL standard is still far away. As it stands, there are some proposed standards that show promise but none that are easy to use and apply to multiple AI problems/use cases. This lack of AI standards can contribute to a lack of AI transparency, and there is no AI algorithm that is 100 percent correct all the time. In other words, we may not understand exactly why AI made a particular decision or arrived at a specific result. This is problematic in search, as if we don’t know why the system arrived at a particular result, we can’t tune or change the configuration. If the wrong product is displayed, that means a lost business opportunity. Businesses need a way to not only understand search results but to tune and validate a specific query for relevance to their specific brand. Ideally, non-technical business owners would be able to see why their transparent AI ranked results the way it did. Meanwhile, these business owners would be able to accept, reject and/or overwrite the AI suggestion based on user behavior. There is no one, standard way to implement AI. The first step to building an AI-powered search solution is having a clear definition of the problem you want to solve. One solution may map to one problem, but not another – and that requires you to redo the experimentation process. Netflix, for example, developed a specific algorithm (through extensive resources and a large volume of data) that has been optimized for one specific problem (recommending specific TV shows). Netflix can continue to optimize this algorithm again and again with new customers. Companies can also buy an off-the-shelf solution that contains existing software that includes AI techniques for a specific problem (an HR solution for analyzing job candidate resumes, for example). The challenge is to decide if your AI search problem requires a customized, off-the-shelf or hybrid approach (more on that later). AI is not an automatic “cure-all” and creating an AI-powered search solution requires extensive testing, experimentation, and evolution. When customers search a brand’s site, it’s more of a journey to buy rather than a question-answer transaction. One specific query can mean different things depending on the context, situation, and user. Normally, the more time a customer spends to find their desired product indicates poor relevance, while better relevance should yield the perfect result instantly. When we think about discovery, we don’t look at the same metric. We look more at the customer’s interaction with the product – and how the customer arrived at their final choice. We can propose a select number of items we think a customer would like, and the more personalized to the user, location, and device, the better. She may then click on two items before making a final purchase. Ideally, we would propose one item if we were certain it would trigger the desired action. What is needed for AI to compute a relevant answer (or potential answers) for the customer’s discovery experience, as well as make further relevant recommendations of complementary products or accessories? Many signals and a feedback loop. We need to take into account the customer’s behavior and individual actions. This should power data enrichment (continually cleaning, enhancing, and updating data), which gives us a more complete view of a customer. And this, in turn, maintains an ongoing, real-time feedback loop with the customer that fuels AI-powered search and query understanding. However, each piece of this puzzle requires different AI tools; no one technique solves everything at once, all the time. AI-powered search is a constantly changing work in progress, as customer behavior constantly changes. Considering the above potential barriers to AI-powered search, how can companies begin to address these issues? How can companies evaluate which type of AI implementation may be right for their business? The first step involves identifying a small set of problems specific to the business that AI can solve. From there, we can decide if buying off-the-shelf software is best, if we should build our own solution in-house, or if we should take a “build and buy” approach. In that scenario, we can build only that part of the solution that is unique for the business. Here, the growing popularity of API-based solutions can make all the difference for developers. APIs meet developers where they are, allowing them to “buy to build faster” by reducing the number of back-end processes and allowing them to get back to the building, experimenting, and iterating. Ultimately, we should have one objective: To provide a digital experience that guides customers to the right information when, where, and how they need it. Author: Julien Lemoine, Co-founder & CTO at Algolia