#51 To RAG or Not to RAG?

As larger context windows become available, why do we still need RAG??

Good morning, AI enthusiasts! Just wanted to share something you might find helpful. From Thursday to Sunday, we’re running a Black Friday sale with 15% off all courses and resources at Towards AI Academy.

If you’ve been thinking about exploring our Beginner to Advanced LLM Developer Course or picking up the Building LLMs for Production ebook, now’s a great time to jump in. Use the code blackfriday_2024 to grab the discount.

If you’ve been waiting for the right time to learn or build, this might be it. Check out the courses here!?

We also have several practical tutorials on building RAG systems, Multi-Agent AI Systems, LLMs from Scratch, and more! Enjoy the read :)

What’s AI Weekly

Many people say that RAG is dead now that we see all the new models coming out with large context windows, like GPT-4o Mini, which can process up to 128,000 input tokens, or, worse, Gemini 1.5 Pro, which can process 2 million tokens. For context, 2 million tokens are equivalent to 3,000 pages. So, do we still need to do retrieval-augmented generation, knowing that better models will continue to emerge in the short and long term with an increased context window and capabilities? This week, I dive into exactly this and help you understand the benefits and trade-offs of using large context models compared to building a RAG pipeline, so you know when and why to spend time and resources developing one. Read the complete article here!?

— Louis-Fran?ois Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community section!

AI poll of the week!

It’s clear the community is highly interested in practical and strategic applications of AI. I would love to know what is the first project or application you'd like to build with the skills you voted for? Share it in the thread!?

Collaboration Opportunities?

The Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too—we share cool opportunities every week!?

1. Shreesha1573 is looking for teammates for a Kaggle competition with an understanding of RAG. If you are available this month, connect in the thread!

2. Lazybutlearning_44405 is looking for a study partner who wants to learn through practical projects using the Python framework. If you prefer this learning approach, reach out in the thread!

Meme of the week!

Meme shared by idkwhattocallmyselfsomeonehelpme

TAI Curated section

Article of the week

Building Multi-Agent AI Systems From Scratch: OpenAI vs. Ollama By Isuru Lakshan Ekanayaka

This article details the creation of two multi-agent AI systems using Python, one with OpenAI's GPT-4 and the other with Ollama's LLaMA 3.2:3b. Both systems, built without pre-existing frameworks, feature agents for summarizing text, writing articles, and data sanitization, each paired with a validator agent. The OpenAI system uses a Streamlit interface, while the Ollama system offers open-source flexibility. It provides code examples, architecture diagrams, and installation instructions, allowing you to understand the fundamentals of building such systems. Each system's modular design promotes maintainability and includes robust logging for debugging.?

Our must-read articles

1. Building a Multi-Modal Retrieval-Augmented Generation (RAG) System: A Comprehensive Guide By Anoop Maurya

This article details building a multi-modal Retrieval-Augmented Generation (RAG) system. This system leverages the ColPali framework for efficient document retrieval using Vision Language Models (VLMs), bypassing traditional OCR. Byaldi, an open-source library, simplifies this ColPali integration. The system incorporates the Qwen2-VL-7B-Instruct model for advanced multi-modal AI capabilities, including video processing and multilingual support. A step-by-step implementation using Streamlit demonstrates PDF upload, indexing, querying, and visual result presentation. It claims the resulting system offers a powerful solution for retrieving information across various media types.

2. Build the Smallest LLM From Scratch With PyTorch (And Generate Pokémon Names!) By Tapan Babbar

This article explains building a small, character-level language model using PyTorch to generate Pokémon-style names. It explains the process step-by-step, starting with creating a character mapping and building a dataset of 801 Pokémon names. A simple neural network with three layers is trained to predict the next character in a sequence, leveraging a context window of three preceding characters. The trained model generates new names by probabilistically selecting characters based on learned patterns. It concludes by suggesting potential improvements, such as adjusting the learning rate, preventing overfitting, and expanding the dataset for enhanced performance.

3. Diffusion Models — A Beginner’s Guide to Math Behind Stable Diffusion and Dall-e! By Shashwat (Shane) Gupta

This article provides a mathematical overview of diffusion models, explaining their underlying principles and how image generation occurs in models like DALL-E and Stable Diffusion. It contrasts diffusion models with other generative models, highlighting the advantages of diffusion models. It explores two key perspectives: the Markov chain perspective, which describes adding and removing noise, and the Langevin dynamics perspective, which focuses on noise-conditioned score generation and facilitates conditional image generation. It also examines architectural choices, training methodologies, sampling algorithms, conditioning methods (such as classifier-guided, classifier-free guidance, and ControlNets), and several improvements, including variance scheduling and latent diffusion.

If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.