I have always been interested in working on how search really works in depth, and this is a part of me trying to understand it in depth so that it can help me or, let’s say, somebody else also go through search as a beginner. But more importantly, it’s like a brain dump through which I’m going to get more clarity on the best way we can design a good search system.
The normal process of working with any kind of query expansion in a search engine right now is that you basically create a big synonym list that actually helps you expand the user query based on the words that are available, by including the synonyms from the list. The synonym list expands the query. It is one step in the direction of solving query disambiguation, but on its own, in mature systems, it can create more problems than solutions. And usually what happens is people start with a big synonym list for the commonly occurring terms of the queries, and then they repeat that list across all types of queries, irrespective of whatever domain or content they have.
The problem is that it sort of becomes an anti-pattern in the longer run as the search engine matures, because when you have more domains, more different kinds of domains and content available, then the synonym list becomes like the wall or the blocker for interpreting the query correctly. In that context, say, for example, a user is trying to search for a wireless mouse and they just search with the keyword “mouse”. Now if your synonym list has the word “rodent”, then everything that is related to that keyword is going to get tagged as a result from the search system. So let’s say a rodent killer will also get tagged, and it will ruin the search results. This kind of term ambiguity, or term meaning changing with context, is the challenge that faces search systems. This kind of problem is called polysemy, where one word can mean different things in different contexts.
And naturally the obvious solution is that you find those broken queries, you see the term that is breaking them, and then expand the synonym list. But the problem is that now it’s an unbounded problem. For how long will you plan to expand these synonym lists? And how much of it will you manage? There is no end to it. So the best way to work on this is to not have a fixed synonym list, because it will then create more problems for you than solutions. The idea is that you have domain-specific synonym lists, which can, let’s say, expand abbreviations or even some sort of terminology that is very common, let’s say from CTO to Chief Technology Officer.
And to solve this, we have to go with a more advanced version of how to disambiguate shorter queries that are not straightforward. It starts with the concept of figuring out the query intent and then figuring out the domain that the intent belongs to, and then having disambiguation logic on top of it, which can improve the knowledge as well to solve a specific query in that relevant context. What are the better ways of solving this problem now that we are in the AI search and LLM era?
Source: AI-Powered Search (Grainger, Turnbull & Irwin), §2.4 and §2.5.1–§2.5.2