Navigating the Landscape of Generative AI: A Beginner's Glossary

In recent months, who and in what technology context hasn't heard of generative AI? Suddenly, a wave of new companies, tools, concepts, and paradigms has captured the attention of every business based on or making use of technology.

Generative Artificial Intelligence (AI) has emerged as a revolutionary field, bringing forth technologies that can create content, imitate human creativity, and enhance various applications. This topic has become the new race among big tech companies and has permeated all strategic decisions of virtually every technology company in 2023.

As everything changes all the time and at an increasingly faster pace, the amount of information and new disciplines we need to learn to keep our careers relevant is growing. Considering this, I decided to write this short guide to help you take advantage of this new wave of innovation provided by generative AI.

If you're just stepping into this world, the concepts and terminology might seem daunting at first. Fear not! This article aims to demystify some of the fundamental concepts that form the backbone of generative AI.

A Brief Introduction

Generative Artificial Intelligence, or Generative AI (in short, Gen AI), has come a long way from the early days of simple rule-based systems in the 20th century. Back then, basic algorithms tried to come up with responses using set rules.

Imagine researchers delving into things like recurrent neural networks (RNNs) and long short-term memory (LSTM) networks. These were the building blocks for fancier generative models. The pivotal moment occurred in 2017 with the release of the paper "Attention is All You Need ," introducing a cutting-edge Transformer architecture. This marked a significant turning point in the field.

Then, things got really exciting when Large Language Models (LLMs) entered the scene. Think of OpenAI's GPT series, starting with GPT-2 and moving on to GPT-3. Trained on massive amounts of data, these models showed an amazing grasp of language details and the knack for creating text that makes sense in context.

As time went on, improvements in model design, training methods, and cool techniques like transfer learning pushed generative AI to even greater heights. The journey through history reveals a constant quest for more powerful, flexible, and human-like generative abilities. This sets the stage for the ongoing evolution of AI-generated content and interactions. It's like AI is on a never-ending quest to become even more awesome!

Concepts

In the upcoming sections, we will explore the fundamental concepts that define the current state of Generative AI. From its core principles to practical applications, each segment will dissect key ideas, providing a clear understanding of the foundational elements shaping the field.

Large Language Models (LLM)

In simple terms, is a powerful computer program designed to understand and generate human-like language. Think of it as a digital language expert on steroids. These generative models, such as OpenAI's GPT (Generative Pre-trained Transformer), have undergone extensive training on vast amounts of text data, enabling them to comprehend context, recognize patterns, and produce coherent text in a remarkably human-like manner. They can be employed for a variety of tasks, such as writing, answering questions, and even generating code. Essentially, LLMs are super-smart linguistic machines that make computers surprisingly adept at understanding and working with human language.

With applications ranging from code generation to creative writing assistance, LLMs are not just changing the way we interact with information, but they are at the center of new AI-driven software development era.

Prompt Engineering

A prompt is like giving instructions to a computer program so it can create what you want. In the context of generative AI, it is the input provided to a generative model to instruct it on the desired output.

These generative models are not deterministic and can be imprecise. Therefore, prompt development or prompt engineering is required to formulate prompts that guide the model in generating good, specific, and relevant content. It involves understanding the model's capabilities and tailoring prompts to achieve the desired outcomes.

Document chunking

Document chunking is like breaking a big piece of writing into smaller parts to make it easier to understand. Imagine you have a long story, and you want to find specific information or summarize it. Document chunking helps by dividing the text into smaller, more manageable pieces.

It plays a crucial role in enhancing the performance of generative AI applications. In the realm of artificial intelligence, especially in tasks like text generation, breaking down large documents into smaller chunks proves invaluable. By dividing a lengthy piece of text into manageable segments, generative AI models can focus on understanding and generating content in a more targeted manner. This approach facilitates more effective learning and processing, as the AI system can analyze and generate responses or creative content in smaller, contextually relevant portions.

Embeddings and Vectors

Word Embeddings is a technique representing words as vectors in a multi-dimensional space, capturing semantic relationships between words. Embeddings allow algorithms to understand the context and meaning of words, not just the keywords used in a keyword search.

Extending this concept, the same principle can be applied to entire sentences (sentences embeddings), enabling models to grasp the context of longer text segments.

Similar to LLMs, there are numerous embedding models available (such as OpenAI's GPT-3 Embedding model , Amazon Bedrock's Titan Embeddings G1 model , and AI21 Studio’s Embeddings model ). These models are responsible for converting text into vectors.

Let's examine the sentence "I just ate an apple. Apple is a big tech.", when utilizing the OpenAI Embeddings Model API, for example, the corresponding vector for this sentence would be: [40, 1120, 30912, 220, 459, 24149, 13, 8325, 374, 264, 2466, 13312, 13].

After embedding the sentence, it's important to observe that the first word, "apple," has a distinct value from the second word, "Apple." The first refers to the fruit, while the second signifies the company. And here is the embedding magic!

For instance, in a keyword search for references to "apple" in a text, any paragraph containing the word "apple" would be retrieved, regardless of whether it pertains to the fruit or the company.

By employing vectors and semantic search, we can specifically look for information about the fruit or the company. In both cases, the semantic context becomes important (as illustrated in the image above). If our focus is on finding information about the company, the search engine will retrieve paragraphs related to Apple Inc.

Therefore, embedding models play a pivotal role in the transformation of text into vectors, streamlining the subsequent storage and retrieval of these vectors within databases.

Vector Databases

These databases are optimized for storing and retrieving vector representations efficiently. They are instrumental in tasks requiring quick similarity searches, such as finding similar pieces of text.

Examples of vector databases include Pinecone , Faiss , Chroma and PostgreSQL with pg_vector extension .

RAG (Retrieval-Augmented Generation):

Large Language Models, however advanced, have a constraint on the maximum prompt size they can handle. This limitation means that longer inputs might need to be trimmed or adjusted to fit within the model's processing capacity. It's like to having a chat with a friend who prefers concise messages – brevity ensures that the model can fully grasp and respond effectively. So, when interacting with LLMs, keep in mind this prompt size limit to optimize the exchange of information.

The Retrieval-Augmented Generation (RAG) model architecture is really useful for getting around the limit on how much you can ask LLMs at once. RAG achieves this by carrying out two steps: retrieval and generation. First, it looks for the important info in big sets of data, and then it only uses that key info when talking to the main model. This way, it avoids using too much information at once and bypass the limit on how much you can ask the model. So, RAG helps us get more info without bumping into the size limit that LLMs have.

Conclusion

As you start learning about generative AI, getting to know these basic ideas will definitely help you understand better. The field is always changing, and keeping up with new ideas and technologies is important to make the most out of generative AI.

So, go ahead, try things out, and let your curiosity lead you through this interesting world of artificial creativity!

Disclaimer: Generative AI helps me to revise and improve this article :)