Architecting Large Language Models

Hey all, Welcome back for the third Episode of Cup of Coffee Series with LLMs. Again we have Mr. Bean with us.

Are you here for the first time ? Check out my first article where I discussed the LLMs intro and transformer architecture. And the second one where I discussed the first two steps involoved in building LLMs.

To sum up the first two steps,

Building a Large Language Model (LLM) starts with a clear vision. The first step is defining your goal - what specific task do you want the LLM to excel at?

Next comes the crucial step of data collection and preprocessing. Here, you gather massive amounts of text relevant to your goal, ensuring it's high-quality and unbiased. This data then undergoes cleaning and formatting.

Let us discuss other steps in detail over here.

3. Model Architecture & Design

The dominant architecture for LLMs is the Transformer. Unlike older models that process text sequentially, the Transformer can analyze all parts of a sentence simultaneously as we discussed in our first article.

While the Transformer is the base, specific design choices are made during LLM development.

Transformer Layers & Hidden Units:

Layers:

The number of encoder and decoder layers in the Transformer architecture determines its capacity to capture complex relationships within the text.

More Layers - Increased complexity so it allows to learn complex patterns but requiring more computational resources and training time

Fewer Layers - limit the model's ability to handle complex tasks but offer faster training and lower computational cost.

Hidden Units

Hidden units are artificial neurons within a Transformer layer. Each unit holds a specific activation value that contributes to the overall output of the layer. The number of hidden units determines the dimensionality of the internal representation used by the model. In simpler terms, it defines the complexity of the information the model can capture within each layer.

Finding the optimal balance between layers and hidden units helps to achieve good performance.

Mr Bean : How we find it ?

Techniques like hyperparameter tuning are used to find this sweet spot for a specific task and dataset.

II) Attention Mechanism Selection:

It is a critical component for understanding relationships between words is the self-attention mechanism. However, there are different ways to calculate the importance of these relationships, each has its own advantages for specific tasks.

Scaled Dot-Product Attention

Scores word relevance based on internal representation similarity (efficient, basic relationships).

Example - Imagine reading a sentence like "The cat sat on the mat." This mechanism would recognize the strong connection between "cat" and "sat" because their internal representations (think of them as simplified meanings) are very similar.

Multi-Head Attention

Focuses on diverse aspects of word relationships simultaneously using multiple "heads" (deeper context understanding).

Example - Think of reading a recipe. One "head" might focus on the ingredients ("flour," "sugar") while another pays attention to the actions ("mix," "bake"). This allows you to understand both what's needed and what to do with them.

Sparse Attention

Reduces computation for long sequences by focusing on a limited set of relevant words.

Example - Imagine skimming a long email. Sparse attention would focus on keywords like "meeting" or "deadline" while ignoring greetings and signatures, helping you grasp the main points quickly.

Universal Attention

Allows attention beyond the current sequence, accessing external knowledge bases for broader context.

Example - While writing a story, you might use a dictionary (like an external knowledge base) to check the meaning of a specific word or ensure a historical event you reference actually happened. This attention mechanism allows the model to access additional information beyond the immediate text.

Choosing the best attention mechanism depends on the specific LLM application and the desired level of complexity.

Mr Bean : Do we use only transformers to design LLMs?

While less common, other architectures are used for specific LLM applications.

Recurrent Neural Networks (RNNs)

These process text sequentially, making them suitable for tasks where order matters, like machine translation. However, they can struggle with long-range dependencies in complex sentences.

Convolutional Neural Networks (CNNs)

Primarily used for image recognition, they can be adapted for text with specific feature extraction tasks, like sentiment analysis.

But Transformers are the leading architecture for LLMs due to their impressive performance. However, other approaches exist for specific tasks, and the future of LLM design may involve further innovation and exploration of new architectures.

For today, we have discussed Architecture and Design step of building LLMs. Thanks Mr. Bean for joining me today. Let us discuss more on our next discussion after 48 hours.

Bye Everyone, Stay Tuned.

Signing off,

Kiruthika Subramani.