Navigating the GenAI Frontier: Transformers, GPT, and the Path to Accelerated Innovation

Deep down, many of us have a lingering fear from sci-fi movies: what if AI becomes self-aware and takes over? In addition, we ask: What if AI replaces my job entirely? Let’s put those fears aside for a moment. A recent statement by Omar Sultan Al Olama, Minister of State for Artificial Intelligence in the UAE, illustrates the transformative power of AI: “If you adopt AI in your life, you will be complete; if you don’t, you will be finished; and if you reject it, you will be completely finished.” This doesn’t strike me as a robot takeover; instead, it’s a powerful tool begging to be further harnessed. Transformers and GPT are at the forefront of this revolution, paving the way for a future where AI complements and empowers us all. This article therefore serves as your guide to understanding these advancements and their potential to revolutionize various fields.

Some Historical Context

While transformers are currently dominating the field with their versatility (multimodality), massive language capabilities (LLMs), and efficient processing (parallel computing), it’s worth remembering their origins. They come from a rich history of neural network architectures. In 2013 and prior, artificial neural networks (ANN), convolutional neural networks (CNN), and recurrent neural networks (RNN) became popular as they handled tasks involving tabular, text, and image data well enough. While effective, RNNs struggled with long text sequences, and ANNs couldn’t handle variable-length data or capture sequential relationships. There had to be a better way. The need for a solution led to sequence-to-sequence learning and the attention mechanism.

Sequence to Sequence Learning with Neural?Networks

A paper published by Google in 2014 tackled the limitations of existing architectures in two key ways:

1. Encoding variable-length sequences into a fixed-size vector

The paper proposed the use of a specific type of RNN called the Long Short-Term Memory Network (LSTM) to handle long-term dependencies. Here, the encoder LSTM takes a variable-length input sequence processes it element by element, captures the relevant information, and combines it with the information from the previous elements. The result is a fixed-size vector that encapsulates the meaning of the entire input sequence.

2. Decoding the fixed-size vector into a new sequence

The decoder LSTM unit now builds the output sequence one element at a time. It starts with the compressed meaning from the fixed-size vector/encoder and a special “start” signal. Then, at each step, it considers both this compressed meaning and the elements generated, to predict the next element in the final sequence. The decoding process continues until it predicts an “end” signal, simplifying the complete output sequence.

While LSTMs improved upon earlier networks it suffered a bottleneck. Their single fixed-length vector for variable-length inputs led to information loss, prompting further research.

Neural Machine Translation by Joint Learning to Align and Translate

The 2015 paper refined the sequence-to-sequence models by introducing attention. It ditched the limited final vector and allowed the decoder to focus on relevant parts of the input sentence at each step. Attention assigns weights to input elements, indicating their importance for predicting the current target word. This way, the model considers both past translations and weighted context from the input, enabling dynamic focus and more effective translations. We can understand this better with an example.

• Input (English): “I want a red apple.”
• Output: target sentence (French): “Je veux une pomme rouge.” (Literally, I want an apple red.)

Without attention (Seq-2-Seq approach):

1. The encoder processes the entire English sentence and creates a single vector.
2. The decoder, relying solely on this vector, might translate it word-for-word, resulting in the grammatically incorrect French sentence “Je veux une rouge pomme.”

With attention:

1. The attention mechanism allows the decoder to focus on relevant parts of the English sentence as it builds the French sentence.
2. When generating the color word in French (“rouge”), the attention mechanism assigns a higher weight to “red” in the English sentence compared to other words.
3. The decoder uses this focus to correctly place the adjective “rouge” after the noun “pomme,” resulting in the accurate translation “Je veux une pomme rouge.”

In this example, attention helps the model understand that “red” describes the “apple” and not the verb “want,” leading to a more natural-sounding French translation.

Once more, a problem surfaces. The architecture relies on LSTM units, therefore adopting the sequential nature of training where only one token can be processed at a time. This led to slow training times and made it impractical to train models efficiently on large datasets. To solve this, transformers came into the limelight.

Transformers (Paper: Attention is all you?need)

To overcome limitations in existing architectures, Google introduced a novel architecture called Transformers. The Transformer architecture eradicated the need for LSTMs by introducing a self-attention mechanism in both the encoder and decoder, allowing the model to consider all elements in the sequences simultaneously instead of sequentially. But why exactly are transformers popular?

Why transformers?

Transformers have become a powerful tool for several reasons:

• Parallel processing: They can be trained on multiple machines simultaneously, significantly speeding up the process.
• Transfer learning: Knowledge gained from one task can be applied to others, improving efficiency.
• Multimodality: They can handle inputs and outputs in various formats like text, images, or code.
• Flexibility: The architecture allows for different configurations, including encoder-only models (BERT), decoder-only models (GPT), and encoder-decoder models (T5).

The working of transformer components

At the core of the Transformer architecture are two parts: an encoder and a decoder. Both parts leverage self-attention layers, the transformer’s key innovation, alongside other components to process information.

Encoder (for understanding text)

• Input embedding: converts each word in the input into numerical representations, called embedding.
• Positional encoding: provides information about the position of each word in the sequence.
• Self-attention layer: allows the model to attend to all the elements of the input simultaneously.
• Feed-forward layer: to add non-linear transformations to the input representations and learn more complex relationships between words
• Layer normalization to normalize the output of the self-attention and feed-forward layers to improve training stability (i.e., the ability to produce consistent results with slight variations in the training data)
• Skip/Residual connections: skip connections serve as shortcuts that allow information to bypass certain layers in the network. This shortcut enables the model network to retain important information from previous layers, making it easier for the model to learn and optimize patterns in the data.

Decoder (for generating text)

• Masked Self-Attention Layer: Similar to the encoder’s self-attention, but with a mask to prevent the model from attending to future words in the target sequence (as it’s generating them one by one). This prevents the model from “cheating” by peeking at the entire target sentence during translation.
• Multi-head attention: attention is computed multiple times in parallel with different linear projections to allow the model to focus on different parts of the input sequence simultaneously, enhancing its ability to capture diverse patterns.
• Encoder-Decoder Attention Layer: This layer allows the decoder to attend to the encoded representation of the source sentence. It helps the decoder understand the context of the source sentence while generating the target words.
• Feed Forward Layer and Layer Normalization: Similar to the encoder.
• Output Layer: returns the decoder’s final output to the vocabulary of the target language, predicting the most likely next word in the translation.

It is also important to note that Transformers are trained to solve a specific NLP task called Language Modeling and LLM. Some popular LLMs include BERT, GPT, and T5. In the next section, we will see how the very first GPT was trained.

How is GPT-1 trained from?scratch?

OpenAI, in their 2018 paper “Improving Language Understanding by Generative Pre-Training,” detailed the training process for GPT-1 using two methods: unsupervised pre-training and supervised fine-tuning. Logically, this can be broken down into:

Data:

GPT-1 is pre-trained on the BookCorpus dataset, which contains over 7,000 unique unpublished books from a variety of genres. This allowed the model to learn the statistical relationships between words and how they are used in context.

Task:

• Unlike standard supervised learning, where models are trained on labeled data (e.g., classifying emails as spam or not), GPT-1 uses an unsupervised learning approach.
• The model is trained on a single task: predicting the next word in a sequence.

Model Architecture:

The model architecture used is a multi-layer transformer decoder, which benefits from multi-headed self-attention and position-wise feedforward layers. This architecture allows the model to handle long-term dependencies in text more effectively compared to recurrent neural networks.

Training Process:

1. Sentence Selection: The model is fed a sentence from the training data.
2. Word Embedding: Each word in the sentence is converted into a numerical representation called an embedding.
3. Processing: The model processes the sentence word by word using its internal layers. At each step, it considers the previously processed words and their embeddings to predict the most likely next word in the sequence.
4. Loss Calculation: The model’s prediction is compared to the actual next word in the sentence. A loss function (e.g., cross-entropy) calculates the difference between the prediction and the actual word.
5. Backpropagation: This loss value is then used in backpropagation, an optimization algorithm that adjusts the internal parameters (weights and biases) of the model to minimize the overall prediction error.
6. Iteration: Steps 1–5 are repeated for a large number of sentences throughout the entire training dataset. Over many iterations, the model learns the statistical relationships between words and becomes adept at predicting the next word in a sequence.

GPT-1 was just the tip of the iceberg as we can see with the developments of more advanced models like GPT-4. You will certainly agree with me that AI can be a reliable copilot when you adopt it. So how can AI even further accelerate our innovation?

The path to accelerated Innovation

Models like transformers and GPT pave the way for an efficient path to innovation. GenAI as a whole presents before us a path filled with diverse opportunities where it can be leveraged to accelerate innovative capabilities for the benefit of humanity. Some of these opportunities exist in:

• Drug Discovery: GenAI can be used to personalize medicine by analyzing a patient’s specific genetic makeup to predict the most effective treatment options.
• Material Science: predicting new materials with desired properties and accelerating innovation in clean energy like high-efficiency solar panels and batteries
• Predictive Maintenance: interpreting sensor data from machines to predict when they’re likely to fail. This allows for preventative maintenance, reducing downtime, and saving businesses money.

Conclusion

In this article, we discussed the advancements of GenAI, from traditional neural networks to transformers, and the basics of how GPT-1 was trained. While these architectures are constantly improving, it is a no-brainer to stay abreast of the ongoing developments in the technology and AI industries. Yes, AI can be scary, but do not reject it; it has come to stay. Make the best and most ethical use of it while we keep our fingers crossed and hope the sci-fi predictions don’t come to pass.

Thank you for coming this far with me??. You can also share your thoughts and fears concerning the advancements in the GenAI space. Until next time, you can read up on my language modeling article here.

Thank you Innomatics Research Labs for the internship program and Kanav Bansal for all the expository live sessions.

References

1. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Published as a conference paper at ICLR 2015 NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE.
2. Kanav, B. (2020). Machine_Learning_and_Deep_Learning/Module 9?—?GenAI (LLMs and Prompt Engineering)/2. Intro to Transformers, LLMs and GenAI/transformers_llms_and_genai.ipynb at master · bansalkanav/Machine_Learning_and_Deep_Learning. GitHub. https://rb.gy/3z7duf
3. Navigating the GenAI Frontier: A UAE Perspective. (2023). Www.youtube.com. https://www.youtube.com/watch?v=sWE4hX2pPOc
4. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
5. Sutskever, I., Oriol, V., & Quoc, V. L. (2014). Sequence to Sequence Learning with Neural Networks.
6. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, ?., & Polosukhin, I. (2017). Attention Is All You Need. https://arxiv.org/pdf/1706.03762.pdf