AI for Query Understanding
In the past decade, the incredible progress in word embeddings and deep learning has fueled an interest in neural information retrieval. An increasing number of folks believe that it’s time to retire the traditional inverted indexes (aka posting lists) that search engines use for retrieval and ranking.
In its place, they advocate a model where search engines use neural networks to represent documents and queries as vectors, and then use nearest neighbor search — or more sophisticated ranking models — to retrieve and rank results.
This revolutionary approach is tempting, but — in my view — misdirected. As I argue below —?and?as?you?can?learn?how to?implement?here —?the right place to focus AI efforts is query understanding.
The traditional information retrieval approach is limiting.
The neural retrieval approach is tempting, because it addresses some of the most frustrating limitations of the traditional information retrieval in general, and inverted indexes in particular.
The traditional retrieval and ranking approach uses a variety of factors, such as BM25, to ensure precision. These factors tend to be highly sensitive to individual keywords. For example, a search for “iphone cases” typically returns slightly different results than a search for “iphone case” — even though the search intents are almost certainly identical.
Similarly, traditional query expansion methods, such as stemming and synonyms, as well as query relaxation, tend to take the query too literally because they focus on keywords rather than the overall query intent. As a result, they can make egregious errors, such as returning iPhone cases when someone performs a search for a “case of apples”.
Embeddings and deep learning allow us to holistically represent and target intent, rather than relying on a reductionist approach based on keywords.
领英推荐
But applying this representation directly to ranking and retrieval skips a critical step: query understanding.
The best place to address query intent is query understanding.
Query understanding is what happens before the search engine retrieves and ranks results: it comprises the searcher’s process of expressing an intent as a query and the search engine’s process of determining that intent. Query understanding involves a combination of holistic and reductionist steps, the holistic steps considering the query as a whole, and the reductist steps breaking the query down into separately analyable parts.
Holistic query understanding with embeddings and deep learning is a great way to look beyond keywords and recognize queries that represent similar or equivalent intent. Before retrieving and ranking results, query understanding can map the query to a representation that canonicalizes the class of equivalent queries. Doing so often allows the search engine to aggregate behavioral signals, such as clicks and purchases, that would otherwise be fragmented among the various expressions of a given intent. Holistic query understanding can also classify the query into a topic or product taxonomy.
Reductionist query understanding can also be valuable. In particular, recognizing entities in queries leads to a richer representation of the query than treating the query as a sequence of keywords. Not only can entity recognition can drive query scoping to improve precision (e.g., recognizing whether “apple” is a brand or product type), but it can also provide context to intelligently expand or relax entities to increase recall (so that we don’t confuse iPhone cases with cases of apples).
Use AI intelligently.
Embeddings and neural networks are powerful tools, and we are only beginning to realize their potential to revolutionize search. But search is more than retrieval and ranking. Search starts with users expressing their intent through queries. Hence, the highest-leverage way to improve search is to through better query understanding.
So let’s focus our AI efforts on this critical part of the search stack. By not taking queries so literally, we will literally revolutionize search!
Senior Director of AI Engineering at Algolia
3 年This paper is what convinced me to stay away from dense search in most cases (as it’s been mentioned, depending on the requirements and the scalability, this might change): https://arxiv.org/pdf/1904.09171.pdf In most cases, really understanding the query is the most critical step or bottleneck, and I don’t think we should hope we can bypass it using dense search only. Now, using dense search after we’ve understood the query, that’s another story but I feel we’re not there yet.
?? Delightful Discovery Experiences ?? | Product Development and Engineering Partner (Search / Discovery / Relevance / GenAI)
3 年Among other things, an effective dense (first-phase) retriever should promote recall ("expansionist" / surfacing true positives which may have received false negative treatment from bm25). To Daniel's point, query understanding is often used to drive filtering/pruning (via domain/behavioral knowledge), rewriting filters into unfiltered queries as a ("reductionist") method of improving precision... and in this case I can see how the two approaches could work against each other. However, incorporating signal *boosts* from query understanding as a second-phase reranker, seems like a complementary application of the strengths of the two approaches for existing pipelines with a (precision-focused) QU step. And to JKB's point, as training techniques improve, the hope is that much of the domain nuance from QU can be embedded in future dense retriever models to shorten the query pipeline and decrease latency.
Retrieval Evangelist
3 年I don’t understand your point. Using a trained dense retriever instead of sparse (bm25) does not rule out using query understanding/classification.?