No Jargon Explanation of Embedding Models with practical use cases

Understanding the cognitive process of the human mind can shed light on how Gen AI works. GenAI is sort of an early stage replica of our very sophisticated mind. Slowing down to think more about how we perceive reality and language is tremendously insightful to understand some of the GenAI concepts.??

When the human mind encounters a question we process it with words. To construct a response, we choose more words from the language to respond with relevance to the question. This is sort of the premise with LLMs in GenAI.?

In this short blog I write about the role of embeddings or embedding models in RAG Apps.?

To learn to learn more about RAG Apps at a high level refer my previous blog

5 Types of LLM Apps - Understanding AI in a Nuanced Way

This is highly simplified so someone who may be non-technical can get this.? Some of the analogies I propose here may not work in all circumstances.

Have you ever played pick the odd one out? Consider the below statements.?

1. Pizza is my cheat meal?
2. Indian food is very flavorful.?
3. I like dining at restaurants with fine ambience?
4. Woody Allen movies are always waxing philosophical.?

When we scan the above four sentences, we find the fourth one to be the odd one.?

Let's break down how we made that determination. Firstly, It was not the words themselves that made us think that the fourth one is the odd one. Meaning, we didn't necessarily run a keyword search in our brain to look for common words between the sentences to figure out the odd one. Instead, what mattered to us was the context.?

When we read each statement, we tried to place them in a context. When we read the first sentence, it made us think we are dealing with a sentence related to food. The second sentence is directed more towards taste in addition to food. We may find the two statements relevant on a scale like this.? This utilizes 2 Dimensions

The third statement appears to slightly throw us off since it deals with the type of place rather than the food itself. When you initially encounter the third sentence it appears to be the odd one. Since our mind has placed the first two sentences in a certain context related to food when we encounter the third as it relates to place, it initially seems like an odd one. One can represent it on a scale like this. This introduces a third dimension.

However, when we encounter the fourth, whoa!!.? We realize that the fourth is the actual odd one.?

In identifying the fourth sentence as an odd one, our mind sort of played the game of finding “similarity” and “closeness” of each of those statements.?

This is exactly what an embedding model does in Gen AI on a given set of big data. Big data is a fancy word for data that may include text, images, videos, pdf files etc.? Embedding models can determine how ‘close’ or how ‘far’ a subset of data may be within the given larger set.?

The role of LLM in this process is to provide the vocabulary required to think. ?

Going back to our example, it would be impossible for someone to figure out the odd one out if someone didn't know the English language. That language role is precisely the role that an LLM plays in this context.

To get slightly technical, embeddings are numerical representations of the sentences in space with dimensions. These mathematical representations are also called vectors. The bottom line is that embeddings help you find the relatedness or closeness of data with each other.?

What are some practical use cases for these embeddings? Embeddings are heavily used in building RAG Apps.

1. Comparing and ranking results by some scale of relevancy
2. Grouping or clustering of customers -? Segmentation?
3. Recommendation of similar items based on preferences?
4. Deal qualification for investment by way of? providing Gen AI some prerequisite for deal consideration

Recently I did a Livestream with Mark Hennings an Inc 500 Entrepreneur who is now the founder of entrypointai.com If you would like to check out the episode here is the link. Also, consider subscribing to the Get in the mode youtube channel where I talk about Self Mastery & Innovation.

https://www.youtube.com/live/nHpun6iAfs4?si=H16Biuvg7jBQ_q3h