A Kitchen Analogy for AI

I've been working through a coursera course on AI. Specifically Langchain: Chat with your data. And one of my main hindrances to understanding the technology was a lack of a generalised structure to AI. Or at least a metaphor. And since I'm well known for using terrible metaphors I decided to outsource the metaphor generation function to, you guessed it ChatGPT.

At first I didn't realise I needed a metaphor. I was just asking questions. So here's how the conversation started:

Me: Give me a short description of Retrieval Augmented Generation        

Retrieval-Augmented Generation (RAG) is a technique in natural language processing that combines the retrieval of relevant information from a vector store, database or a set of documents with the generation of text based on that retrieved information.

It typically employs a two-step process: first, a retrieval system identifies relevant text segments or documents from a corpus, and then a generative model produces text based on both the retrieved information and the input prompt.

This approach allows the model to utilize external knowledge to produce more informed and accurate responses.

There's still a few things in there I had heard used and not grokked. Certainly there were more questions to be asked. The next question filled the picture a bit more.

Me: What is the difference between a large language model and a vector store?        

Large Language Models (LLMs) and vector stores serve different purposes and have different underlying mechanisms, although they can complement each other in certain applications:

1. Nature and Purpose:

- Large Language Models (LLMs): LLMs are designed to understand and generate human-like text based on the input they receive. They are trained on vast amounts of text data and are capable of tasks such as translation, summarization, question-answering, and many others.

- Vector Stores: Vector stores are data structures or systems used for storing and retrieving vectors efficiently. They are often used in similarity search applications where the goal is to find the vectors in the store that are most similar to a query vector.

2. Underlying Mechanisms:

- LLMs: The core of LLMs is a neural network architecture, often transformer-based, which learns representations of text during training and can generate or classify text during inference.

- Vector Stores: These are more about efficient data storage and retrieval, and do not have a learning mechanism inherent to them. The vectors stored in a vector store might be produced by some other process, possibly even an LLM.

3. Usage in Systems:

- Combining LLMs and Vector Stores: In some advanced systems, LLMs and vector stores might be used together. For instance, an LLM might generate vectors representing pieces of text, which are then stored in a vector store for efficient retrieval later on. This combination can be particularly powerful in retrieval-augmented generation systems, where an LLM generates text based on information retrieved from a vector store.

4. Learning and Adaptation:

- LLMs: They have the ability to learn from data and adapt to new information to some extent, especially during the training phase, and can generalize to handle a wide range of inputs.

- Vector Stores: They don’t learn or adapt; they just store and retrieve data based on predefined or externally computed metrics. In essence, LLMs are about understanding and generating text, while vector stores are about efficiently storing and retrieving vectorized data.

This I found very useful. I could now understand that there was a relationship between vector stores and LLMs.

Now, I also knew that there were many different types of LLM, that there was Retrieval Augmented Generation (AI twitter shortens this to RAG) as well as Pure Generative Models. A vector store belongs with the former. Sidenote there are many different types of Chefs (Supervised Fine-Tuning, Memory Networks, and Knowledge-Integrated Models) but perhaps that's for another day.

However I still couldn't intuit the relationship between the vector store and the Large Language Model. So, I asked ChatGPT for an analogy and we went through a few rounds of refinment of the analogy before I ended up with something that I was happy with.

The goal of my conversation was to build an analogy to what was happening between gathering data, and presenting a result to a user. It was important that ChatGPT also included the technologies used (I am aware of tensorflow for example but I don't know how its used or to what end ). As it turns out I originally asked for a kitchen analogy with a focus on data loading, and then asked for a description of the full system.

The Vector Store Focused Analogy

Imagine a large, bustling kitchen where there is a dedicated prep team and a talented chef. The prep team represents the preprocessing steps or other components of the system, while the chef represents the LLM.

1. Preparation: The prep team (preprocessing components) is responsible for sourcing, chopping, and organizing the ingredients. They vectorize and embed the necessary data, then carefully store these prepared ingredients in a well-organized pantry (the Vector Store).

a. Vectorization:

- Analogy: Imagine each ingredient has unique flavors and nutritional profiles. The prep team analyzes each ingredient and creates a detailed label that describes its flavor profile, nutritional content, and possible culinary uses. This label acts as a summary or a fingerprint of the ingredient, encapsulating its essence in a way that can be quickly understood and compared.

- Technical: In the embedding process, text segments are converted into numerical vectors which capture the semantic essence of the text. This process, often known as vectorization, transforms the text into a format that can be efficiently compared and analyzed.

b. Dimensionality Reduction:

- Analogy: The labels created by the prep team are concise, retaining only the most important and distinguishing information about each ingredient while omitting less relevant details. This concise representation ensures that the chef can quickly grasp the essence of each ingredient without getting bogged down in minutiae.

- Technical: Dimensionality reduction is often a part of the embedding process where the high-dimensional data is transformed into a lower-dimensional space, retaining only the most salient features. This makes the data more manageable and speeds up the retrieval process.

c. Indexing:

- Analogy: Once labeled, the ingredients are organized in the pantry in a way that makes sense - perhaps grouped by food groups, flavors, or other relevant categories. This organization creates an index that the chef can use to swiftly locate and retrieve the ingredients needed for a particular dish.

- Technical: The vectors created during the embedding process are often indexed for efficient retrieval. Indexing structures like KD-trees or hashing techniques may be used to ensure that the retrieval process is fast and accurate.

2. Retrieval: When it's time to cook, the chef (LLM) goes to the pantry to gather the ingredients prepared by the prep team. Similarly, given a query, the relevant vectors are retrieved from the vector store for use by the LLM.

3. Creation: With the retrieved ingredients in hand, the chef crafts a dish according to the customer's request. In the same vein, the LLM, with the retrieved vectors, generates a response or performs the task at hand based on the input query.

This analogy helps to clarify the roles and interactions between the preprocessing components, the LLM, and the vector store in a system.

The Full Kitchen Analogy

Let’s put together the full kitchen analogy, incorporating all the elements we’ve discussed, including the knowledge graph:

The Kitchen (AI System):

- This is where all the cooking (processing and generating responses) happens. It includes the layout, tools, and equipment (TensorFlow, PyTorch, Keras) that chefs (machine learning and deep learning models) use to prepare dishes (outputs).

The Chefs (Machine Learning and Deep Learning Models):

- Chefs with varying levels of expertise (ML for general techniques, DL for more complex tasks) work here. They use their training and skills (algorithms and computations) to prepare meals (solutions) for the restaurant's patrons (users).

The Prep Team (Data Preprocessing and Vectorization):

- This team gets the ingredients ready (data preprocessing) by cleaning and chopping them (vectorization). They ensure ingredients are ready for the chefs to use in cooking (model input) (scikit-learn, NLTK, SpaCy).

The Pantry (Vector Store):

- A well-organized space where ingredients (data in vector form) are stored (FAISS, Elasticsearch, Milvus, Annoy, Pinecone, Weaviate). Chefs retrieve these ingredients when they have a specific dish (query) to prepare, thanks to the efficient organization by the prep team (embedding).

The Interactive Recipe Book (Knowledge Graph):

- A comprehensive encyclopedia (knowledge graph) provides detailed knowledge about each ingredient and dishes (Google Knowledge Graph, Microsoft Satori, DBpedia, Neo4j, Amazon Neptune, Wikidata). Chefs consult it to understand the intricacies of ingredients and to get inspiration for creating new dishes or tweaking existing ones.

Specialized Cooking Techniques (Model Architectures and Learning Techniques):

- These are akin to specific cooking methods like sous-vide, fermentation, or baking (Supervised Fine-Tuning, Memory Networks, Knowledge-Integrated Models). Chefs use these techniques to create unique dishes or to ensure consistent quality (model fine-tuning and enhancements).

The Ordering System (User Interface):

- Patrons (users) place their orders (queries) through this system, which the chefs use to understand what dishes to prepare (model inference).

The Dish (Generated Output):

- The final meal (output) that is served to the patron (user), crafted using the ingredients, the recipe book, and the chefs' specialized techniques.

In this bustling kitchen (AI system), the chefs (models) are adept at creating a wide array of dishes (outputs), from the simple to the complex, by utilizing a variety of ingredients (data) and cooking methods (algorithms). The well-organized pantry (vector store) and the interactive recipe book (knowledge graph) ensure that they have all the information and resources at their fingertips to satisfy the culinary requests (queries) of their patrons (users) efficiently and effectively.

And while this effort didn't teach me a new skill it gave me something far more valuable, a model which I could refer to when thinking about various parts of an AI architecture. And it's benefits are already apparent. When watching a video on vectorisation I could quickly understand what was being referred to. Right now I'm working on a vector store and I know now that I can focus on three distinct techniques: Vectorisation, Dimensionality Reduction and Indexing. And that there are platforms out there that I can use: FAISS, Elasticsearch, Milvus, Annoy, Pinecone and Weaviate.

In this small experience I can say that while AI may bring many pros and cons to society it will outdo 99% of people in communicating complex ideas through the power of analogy.