CHATGPT HAS NO IDEA ABOUT MY SOURCE OF TRUTH

Latest trends in today's tech-driven world, artificial intelligence is at the forefront of innovation. One of the latest trends in AI is centered around generational models, and you simply can't escape their impact :). In this blog post, I'd like to introduce you to an exciting concept called RAG: Retrieval Augmented Generation. RAG is all about improving the quality of conversational agents (i.e LLMs), by grounding them to a source of knowledge. This, in turn, enhances the LLM's ability to provide accurate and relevant information.

To understand RAG better, let's delve into some essential concepts like Vectors and Embeddings. A vector is essentially an array of numbers. Embeddings, on the other hand, are similar to vectors but serve the purpose of converting text into tokens (vectors). Today, accomplishing this conversion is easily achievable using a function. But why is this important, you ask? Well, let's consider a query, for instance: "As a dynamic meteorologist derive the stability equation for leapfrog approach otherwise called Courant-Friedrichs-Lewy condition?" If we submit this query without any grounding/Source of truth, we might receive an inadequate response due to AI models being trained on vast datasets. RAG's logic comes into play here. By utilizing our source of truth, which contains information relevant to our specific context, and converting it into vectors, we can constrain the response to give us adequate information.

The magic of retrieving the right response happens through a mathematical function called cosine similarity, which measures the similarity between two non-zero vectors in an inner product space. By looping through our vectors and retrieving a similar vector based on the query, we can provide a tailored response. Sounds a bit complicated, Delve in to the bigger picture below :).

For example, our vectors may contain an embedding and associated metadata like this:

- Embedding: [0, 4, 5] -> Array of vectors

- Metadata: {textContent: 'Facts about Paris'} -> namespace

In practice, we can have a function that splits our text into documents, effectively dividing it into manageable chunks.

So, how do we code our text into vectors, and where do we store these vectors for retrieval and cosine similarity calculations? PineConeDB, a database designed for storing vectors. Here's a high-level overview of how this works: When we receive a query, we convert it into a vector (i.e., embed it). Then, we scan through all the vectors in the PineConeDB to find the most similar one. Once we identify the similar vector, we retrieve the original vector from the PineConeDB, complete with its metadata. With this information, we can create a prompt that feeds into ChatGPT to generate a relevant response.

Here are the steps in a nutshell:

1. Obtain your source of truth.

2. Split your source of truth into manageable chunks.

3. Vectorize and embed these chunks.

4. Store the resulting vectors in PineConeDB.

Now, let's talk about retrieval and response generation. At this point, we're essentially searching for the right answer. Here's what we do:

1. Embed the query (convert it into vectors).

2. Query the PineConeDB for similar vectors.

3. Extract the metadata from the similar vector.

4. Feed the metadata into the OpenAI prompt to generate a contextually appropriate response in line with our source of truth.

With an understanding of these concepts, let's explore some packages and tools that make this ambitious project possible.

LangChain: This tool aids in splitting our source of truth into manageable chunks, making the process more efficient.

PineConeDB: This database is used for storing and organizing our vectors, making it easy to access and retrieve the information we need.

OpenAI-Edge: This component allows us to query the database and seamlessly feed the retrieved metadata into our ChatGPT prompt.

Vercel AI SDK: For creating the ChatGPT-like conversational experience, Vercel AI SDK is a recent and powerful SDK that simplifies the process, abstracting away much of the complexity.

The power of AI, specifically generational models like ChatGPT, is undeniable. However, to make these models truly effective and aligned with our source of truth, techniques like RAG and tools like PineConeDB, LangChain, and Vercel AI SDK come into play. With these technologies at our disposal, we can bridge the gap between AI models and real-world information, creating more accurate and contextually relevant conversational agents.

Note: The era of AI-powered content interaction has arrived, and it's only going to get more fascinating from here.