Large Language Model Embeddings Fundamentals

Imagine an intricate web, woven from threads of words and meaning, stretching infinitely across a hidden landscape. This is the world of embeddings in large language models (LLMs). These embeddings act as invisible coordinates, charting relationships between words and concepts, creating a dynamic map where words aren't just symbols, but anchors of meaning.

Think of an LLM as a cartographer of language, mapping out this landscape where similar ideas naturally cluster together, forming regions of related meanings. Words like “happy” and “joy” gravitate near each other, while more complex ideas stretch across different dimensions, capturing subtleties of context and usage. This is where embeddings come in: they assign each word a unique, multidimensional “address,” placing it on this vast map of meaning.

But before we reach these intricate landscapes, there are the basics—Token IDs. Each word begins as a Token ID, a simple identifier. Yet, in isolation, these IDs are like labels without meaning, just signposts without a path. They are the initial sparks, but they need form and structure to build understanding. They can also handle unknown words by using advanced tokenizers like Byte Pair encoding.

This is where token embeddings enter, adding depth and context to each token. With token embeddings, each word or phrase begins to take shape in the model’s “mind,” forming the first hints of relational meaning. It’s as if each word has been placed on a giant web, connected by strings to others that carry similar meanings or contextual ties.

But language isn’t static—it’s fluid, shaped by sequence and flow. To understand this, the model needs positional embeddings. Absolute positional embeddings anchor each word in a fixed place, like coordinates on a map, ensuring that the LLM recognizes whether a word appears at the beginning, middle, or end of a sentence. On the other hand, relative positional embeddings provide flexibility, noting the distance between words to capture meaning even when words move or change order. Think of this as the model gaining an awareness of language’s “rhythm”—the way words relate to each other no matter where they are.

These components — token embeddings, and positional embeddings—come together to create input embeddings. From here, the LLM sets off, with embeddings as its compass and road map, navigating the vast, nuanced landscape of human language. With each layer, it gains a richer understanding, a deeper capacity to predict, generate, and respond.

Embeddings, in essence, are the LLM’s secret power—the hidden architecture behind its “understanding.” They allow it to go beyond mere words, capturing the complex dance of meaning and context that makes language alive. For me, this journey feels akin to mastering physics years ago, where every complex equation revealed a deeper layer of reality. Embeddings, too, unveil a hidden layer of language, guiding the model’s journey from raw symbols to refined understanding.

Below is how the process looks like :

The picture has been taken from the Sebastian Raschka GitHub repo.