How do you choose the best information retrieval model for your domain?

In Vector space model, we consider each word as an independent word from the other words. So, each word is treated as one dimension and document can be represented as a word vector. For example, consider the goat language which can use words like "baabaa", "baaba". In VSM, "baabaa" can be presented as 2b+3a. And it is different from cat language, as cat uses the different sounds. We use cosine similarity to measure the similarity between the documents. Cosine similarity measures angular distance. It does not depend on document length. In information retrieval, cosine similarity measures distance between the document and query. You would retrieve document which has highest similarity score, when matched with query keywords.

The key concept is to move away from sparse vectors (traditionally based on bag of words approaches) to dense vectors (where the vector is the output of deep neural networks trained and fine-tuned for sentence similarity). We've seen new vector search engines sprouting here and there and claiming a revolution but the reality is that the approximate nearest neighbours algorithms are just part of the solution. Finding models that consistently encode text to good vectors is hard, especially in languages that are not that popular. So far all the industrial applications I've seen failed to deliver consistency, it's easy to quickly build a prototype, but going to production is much more time-consuming and explainability is low (bug-fixing is hard

Vectors immediately solve huge problems with keywords retrieval systems: 1. Long sentence keywords like "I need shoes for running in wet conditions" 2. Multilingual queries and content: create your content once and deploy it internationally 3. NLP: typos, synonyms, antonyms Disadvantages may be a lack of precision, which can be resolved by fine-tuning or reranking.

There are many ways to make a Boolean model ???????? sensitive to the choice of terms and it's possible to enhance BM with ranking algorithm. ???????, by 'enhancing' a BM there is a risk of losing it's benefits, like 100% accuracy (???? ???? ???????? ?????????????????? ?????????? ??????????????) or it's simplicity and transparency.

The boolean model is still very useful, especially for the matching phase of the retrieval (identifying the good candidates to be ranked). This is valid for traditional lexical (keyword-based search) or hybrid and allows great flexibility. Most modern systems by the way encourage a less strict approach, so rather than using explicit boolean operators you may just state the "minimum should match" in terms of how many clauses should match for a candidate to be considered good. Even in 2024, I find an expert usage of boolean clauses and minimum should match to be at the basis of well-functioning traditional search engines. Especially when talking about Apache Lucene/Solr, Elasticsearch, OpenSearch and Vespa (open source)

SQL databases are not optimized for natural language query. In Information Retrieval engine, optimization is implemented at search and storage level. Boolean model uses inverted index for storage. Inverted index is a hash-map like data structure, where word is present as hash-map key and hash-map value is the list of documents. Boolean model parses the natural language query and returns the list of tokens. By default, it combines all the query keywords with "and" operations and search them in inverted index. It returns the list of documents that contains query keywords. It also supports "or" and "negation" operations, if these operations are specified explicitly in the query. Search results are generally sorted by relevancy score.

Most of the information retrieval engine, supports query completion or auto-suggest features. Language models can generate query from the underlying indexed documents. The language models is a probabilistic model which uses various statistical techniques for generating a sequence of keywords. Basically, language models supports two search techniques, 1) greedy search b)Beam search. Greedy search chooses the best word with the highest probability for each word position in the generated sentence. Beam search expands the search result to get the N best keywords at each word position. It identifies the N best sequence by considering the probabilities of all the combinations of the word sequence.

I've been consulting about Search for more than a decade, from fortune-100 companies to small startups and it's apparent that there's a huge deficiency in measuring search quality, both offline and online. Many want to improve their search engines but have no methodic idea of how they perform. My recommendation is to start from the basics: identify the most important metrics for your business, closely related to the search engine (don't focus on vague profit or conversion rate, start from the signals on the search result page). Once you collect clicks, downloads, add to favourites etc you can consistently monitor online what's going on and potentially test different retrieval models. Then you can use the collected data offline as well.

Nowadays the average salesman would recommend going with a vector-based dense retrieval model. Vectors coming from sentence similarity fine-tuned large language models are still unreliable on their own at production grade. Don't get me wrong, it's a cool technology and will be the future, but right now my recommendation would be to start simple, with a decently good traditional keyword search. Once you realise you have a vocabulary mismatch problem (terms in the queries don't align with terms in the documents), you have multiple fields involved in the search, with different importance, it may be time for you to integrate more advanced solutions such as Large Language Models for the first problem and Learning To Rank for the second.

It depends on the search users, whether they use short or long phrases for the keywords. But generally speaking, I would opt for a vector search to always get results, followed by a reranking step to align the best results with certain business signals, like clicks or sales. If you do not collect signals, you can rerank from a specially trained LLM Reranker like Cohere's. If you do collect signals, use recommendations filtered by the results ids. We do it with Weaviate results reranked by Recombee's recommendations.