How Do Embeddings Help Reduce Hallucinations?

Hey there! We are super excited to kick start a series of posts on XenAIBlog where we’ll be sharing our learnings from diving deep into the world of Large Language Models (LLMs). We've been on quite the journey, exploring fascinating use cases, and discovering some cool stuff along the way. Today, let’s dive into a key finding that has truly transformed our approach - how embeddings can be a game-changer in dealing with hallucinations when working with LLMs.??

Let's rewind a bit. Now, what is NLP? Natural Language Processing (NLP) is a branch of AI that makes computers understand and generate human language. It's almost like teaching a computer to speak and write like a human.?

Now, within the world of NLP, LLMs are super-smart models that can generate human-like text. They've been trained on massive piles of text data, like books, articles, and web content. But, and it's a BIG but, they're not perfect. Sometimes, they generate text that's not quite right. We call these snags "hallucinations," and they can be a real buzzkill.?

What we realized was that LLMs don't fact-check like humans do. They just generate text based on patterns they've learned from their training data. So, if the training data includes mistakes or contradictions (which, let's face it, is pretty common on the internet), LLMs might generate nonsense or just plain wrong facts.??

So, how did we fix this hallucination problem? One trick in our toolkit was to fine-tune the model using embeddings. Embeddings are like secret codes that represent words, sentences, or whole documents in a way that captures their meaning. Imagine you're playing detective, and you need to describe things in a way that only other detectives would understand. You might come up with secret handshakes or symbols. Well, in the world of NLP, we have embeddings. They're like secret handshakes for words, but in a mathematical form. These embeddings help us understand the meaning and relationships between different pieces of text. So, when we say "cat," the computer knows it's not the same as "dog" because they have different secret handshakes (embeddings).??

We then took our LLM and gave it a little extra training using these embeddings and local data (data that we want the model to learn and answer from). So, when we prompt the model with our question, the embeddings help the model compare the text it is generating with the input prompt and the local documents it was trained on. This extra context was a game-changer.?

But wait, there's more! We didn’t just throw the embeddings into the mix haphazardly. We used something called "vector databases" to manage them efficiently. These databases are a way to store the embeddings and perform operations such as chunking, embedding, and retrieval.?

Chunking involves breaking down a lengthy document into manageable pieces, thus simplifying the task for the computer, whilst managing the issue of token size. Then, we turned those chunks into embeddings using the same technique we used for the LLM.?

Now, the real magic happens during retrieval. This is where we find the most similar chunks to our input prompt using a math function like cosine similarity. It’s like finding puzzle pieces that fit perfectly.?

By using these embeddings and vector databases, LLMs can tap into relevant information from local documents. Think of it as having an experienced research assistant who fetches the right books for you. This helps the LLM generate text that makes sense in the context of the prompt and the documents it's referencing.?

Embeddings offer not just a means to fine-tune your model using local data but also provide a way to incorporate large documents by breaking them down into appropriately sized chunks for the given prompt.??

We hope this article has shed some light on embeddings and vector databases. Stay tuned for next week when we'll dive into an intriguing perspective on the tiktoken tokenizer for LLMs.???