Understanding deep learning models as overcoming limitations of previous models

In the previous newsletter, I shared about my approach to teaching AI using basic maths ideas - and why this approach may be suitable to many.

In this edition, I expand on this idea more by showing how deep learning algorithms can be understood by overcoming limitations of previous models.

Before I proceed:?

1. If you want to meet us, see the Oxford AI Summit ?
2. We have also launched two of our popular courses: low code AI apps - which is for non developers and artificial intelligence with digital twins
3. Finally, I noticed that this newsletter now has close to 90K subscribers. Thanks for your support as always.?

This edition was inspired by a quote from Yoshua Bengio in his book:

Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

source https://www.deeplearningbook.org/

I thought - Can we extend this idea more broadly? I.e.?

Can we explain the evolution of deep learning models from MLP - CNN - RNN/LSTM to transformers by showing how the subsequent models are enhancements of the core deep learning/MLP model

This is great for learning because each new model builds upon the foundational concepts of its predecessors and introduces new features to overcome their limitation - thereby making it easier for the learner..?

So, here is how we can see it

Multilayer Perceptrons (MLP) MLPs are the foundation of neural networks, consisting of one input layer, one or more hidden layers, and one output layer. Each layer is fully connected to the next, and neurons in these layers use a set of weights and biases to learn patterns in data. Because MLPs treat input features as independent, temporal and spatial information is lost. Hence, they are not efficient in recognizing patterns in sequential data like text or time-series.

Convolutional Neural Networks (CNN) introduce convolutional layers that apply filters to the input to detect spatial hierarchies and patterns such as edges and shapes in images. This makes them exceptionally good for tasks like image recognition. See this video to explain these ideas in? less than five minutes. Thus, CNNs use a combination of convolutional layers, pooling layers, and fully connected layers. The convolutional layers automatically learn spatial hierarchies of features from the input images. Pooling layers reduce the spatial size of the representation, reducing the number of parameters and computation in the network.

Recurrent Neural Networks (RNN) ? While CNNs excel in tasks involving spatial data (like images), they are not designed to handle sequential data where the order of the input matters. Enhancement over MLP: RNNs are designed to handle sequential data. They have a memory mechanism that allows them to remember previous inputs in the sequence, making them suitable for tasks like language modeling and time-series prediction.RNNs process sequences by iterating through the sequence elements and maintaining a 'state' that implicitly contains information about the history of all the past elements of the sequence.?

Long Short-Term Memory (LSTM ): LSTM is a special kind of RNN capable of learning long-term dependencies. It introduces gates that regulate the flow of information to be remembered or forgotten - specifically to address the vanishing gradient problem.? Vanishing gradient problem is a phenomenon that occurs during the training of deep neural networks, where the gradients that are used to update the network become extremely small or "vanish" as they are backpropogated from the output layers to the earlier layers. LSTMs address the problem of long term dependencies thereby overcoming the problem of vanishing gradients.?

In RNNs, if the previous state that is influencing the current prediction is not in the recent past, the RNN model may not be able to accurately predict the current state. For example,if we have the sentence - “Alice is allergic to nuts. She can’t eat peanut butter.”?

In this case, the context of nut allergy is significant but if that context was? a few sentences prior, then it would make it difficult, or even impossible, for the RNN to connect the information.?

To remedy this, LSTMs have “cells” in the hidden layers of the neural network, which have three gates–an input gate, an output gate, and a forget gate. These gates control the flow of information which is needed to predict the output in the network.? For example, if gender pronouns, such as “she”, was repeated multiple times in prior sentences, you may exclude that from the cell state.? Example adapted from IBM

Despite this improvement, LSTMs still suffer from issues like vanishing and exploding gradients, making it hard to learn long-range dependencies in very long sequences.

Transformers: Enter Transformers and attention mechanism. To overcome the problem of very long sequences, we can think of transformers processing long sequences in a ‘non sequential’ manner. Instead of using recurrence and convolutions, transformers use attention mechanisms to weigh the significance of different words in a sentence, capturing the context more effectively and efficiently.?

The attention model is a mechanism in neural networks that enables the model to focus on different parts of the input data, much like humans do when we pay attention to specific aspects of our environment.? The fundamental idea behind attention is that not all parts of the input are equally important for a given task. For instance, when translating a sentence, certain words in the source language may have more impact on the translation of a specific word in the target language. Attention models weigh the input features (like words in a sentence) to focus more on the relevant ones and less on the irrelevant ones. In tasks like language translation or text summarization, the meaning of a word can depend heavily on the words around it. Attention models weigh the influence of different parts of the input sequence, which means they can capture the context and nuances of language better than models that process the input sequentially.

The transformer model is based on self-attention mechanisms that weigh the influence of different parts of the input data. It consists of an encoder to process the input and a decoder to produce the output, both of which are composed of multiple layers of self-attention and feed-forward networks.

There are two other innovations in transformers: Firstly, unlike RNNs that process data sequentially, attention mechanisms can process all parts of the input data in parallel, leading to significant improvements in training speed. This is especially beneficial when dealing with long sequences. Secondly, while CNNs and RNNs have a static structure in how they process inputs, attention in Transformers allows the model to dynamically focus on different parts of the input as needed, leading to a more nuanced understanding and better handling of complex relationships in the data.

I hope you find this approach useful.

1. If you want to meet us, see the Oxford AI Summit ?
2. We have also launched two of our popular courses: low code AI apps - which is for non developers and artificial intelligence with digital twins
3. Finally, I noticed that this newsletter now has close to 90K subscribers. Thanks for your support as always.?