Bahdanau Attention Mechanism

In my last NLP post regarding NMT(Neural Machine Translation), I shared about its architecture in a very intuitive manner. I shared about Encoder, Context Vector, and Decoder.

??Post Link

??Article Link(NMT Architecture)

At the end of the article, I said we could add Attention Mechanism to our decoder. So, let's continue from there.

• The next step of our journey takes us to one of the most important concepts in machine learning, 'Attention'.
• So far, the decoder had to rely on the encoder's last state as the 'only' input/signal about the source language. This is like asking to summarize a sentence using a single word. Generally, when doing so, you lose a lot of the meaning and message in this conversion.
• Attention alleviates this problem.

Instead of relying just on the encoder's last state, attention enables the decoder to analyze the complete history of the encoder's state outputs. The decoder does this at every step of the prediction and creates a weighted average of all the state outputs depending on what it needs to produce at that step.

For example, in the translation from English to German, I went to the shop -> ich ging zum Laden, when predicting the word ging, the decoder will pay more attention to the first part of the English sentence than the latter.

?? The context/thought vector is a performance bottleneck

As we have seen in the encoder-decoder architecture of NMT, the encoder part spits out a summarized representation of the source language sentence as 'context/thought vector', which basically creates a link b/w the encoder and the decoder, which later the decoder uses to translate the sentence.

• To understand why the context/thought vector is a performance bottleneck,Let's imagine translating the following English sentence:

I went to the flower market to buy some flowers

This translates to the following:

Ich ging zum Blumenmarkt, um Blumen zu kaufen

• If we are to compress this into a fixed-length vector, the resulting vector needs to contain these:

1. Information about the subject (I)

2. Information about the verbs (buy and went)

3. Information about the objects (flowers and flower market)

4. Interaction of the subjects, verbs, and objects with each other in the sentence

• Generally, the context vector has a size of 128 or 256 elements. Reliance on the context vector to store all this information with a small-sized vector is very impractical and an extremely difficult requirement for the system.
• Therefore, most of the time, the context vector fails to provide the complete information required to make a good translation.
• This results in an underperforming decoder that suboptimally translates a sentence.
• To make the problem worse, during decoding the context vector is observed only in the beginning. Thereafter, the decoder GRU must memorize the context vector until the end of the translation. This becomes more and more difficult for long sentences.

?? How does Attention deal with this issue?

Attention sidesteps this issue:

1. With attention, the decoder will have access to the full state history of the encoder for each decoding time step.

• This allows the decoder to access a very rich representation of the source sentence.

1. Furthermore, the attention mechanism introduces a softmax layer that allows the decoder to calculate a weighted mean of the past observed encoder states, which will be used as the context vector for the decoder.

• This allows the decoder to pay different amounts of attention to different words at different decoding steps.

Conceptual Breakdown of the Attention Mechanism:

?? The Bahdanau Attention Mechanism

(Also called Additive Attention)

The Bahdanau attention mechanism is introduced in the paper Neural Machine Translation by Learning to Jointly Align and Translate , by Dzmitry Bahdanau .

The attention mechanism was introduced to address the bottleneck problem that arises with the use of a fixed-length encoding vector, where the decoder would have limited access to the information provided by the input. This is thought to become especially problematic for long and/or complex sequences, where the dimensionality of their representation would be forced to be the same as for shorter or simpler sequences.

? Simple & Intuitive Explanation of Bahdanau Attention:

• The Bahdanau Attention Mechanism, also known as Additive Attention, is like a spotlight that helps a machine learning model focus on the most relevant parts of a long piece of information when making decisions, just like how you pay attention to different words when reading a sentence.

Here's a simple and intuitive explanation:

• Imagine you're translating a sentence from one language to another, and the sentence is quite long. Bahdanau Attention is like having a little assistant who highlights specific words in the original sentence for you as you translate.

1. The Sentence: Let's say you have a long sentence in a foreign language you want to translate, like "The big blue car drove quickly down the winding mountain road."
2. The Assistant: Your Bahdanau Attention assistant looks at each word in the sentence and decides which words are the most important for you to pay attention to while translating.
3. Highlighting: It highlights certain words, like "big," "blue," and "car," which are the key pieces of information for understanding the sentence.
4. Translating: As you translate, you focus more on the highlighted words, so you might say something like, "The important thing here is that there's a big blue car." You give extra importance to those highlighted words because they carry the crucial details.
5. Dynamic Attention: What's cool is that the assistant can change its highlights for different sentences. If the next sentence is, "The small red bicycle went slowly up the steep hill," it will highlight different words like "small," "red," and "bicycle."

In summary, Bahdanau Attention is like having a helpful spotlight that guides you through understanding and translating sentences by emphasizing the important words. It's a way for machines to focus on the relevant parts of information when processing sequences of data, making them more efficient and accurate in tasks like translation, summarization, and more.

Other resources to learn about Bahdanau Attention:

• Machine Learning Mastery : The Bahdanau Attention Mechanism
• d2l.ai: The Bahdanau Attention Mechanism

BTW, if you are interested in learning more about this, here is my very in-depth notebook on this topic, explaining the concepts and code implementation in great detail.

??GitHub Link: Seq2Seq Learning - Implementing NMT System