Talking to Computers: A Peek into Word Embeddings ????

When we talk to computers, we've got to speak their language, and they only understand numbers. Imagine if every letter in the alphabet had a number tied to it, like A is 1, B is 2, all the way to Z. Now when you type something, the computer turns each letter into its number buddy. This is similar to word embedding. While this character-to-number mapping is a helpful starting point, true word embeddings involve representing words as vectors of real numbers to capture more complex relationships.

The picture above shows exactly that. It's like a secret code where each letter matches a number. So, if you wanted to tell a computer about an apple, it would change 'apple' into a bunch of numbers using this code. That's the first step in getting a computer to understand what we're saying or to help it figure out what to say back. Simple as that!

1. One-Hot Encoding - Simplifying Words for Computers

Now let's take it a step further. We've got our numbers for letters, right? But what about whole words? Computers need a way to understand those too.

This is where One-Hot Encoding (OHE) comes into play. It's a bit like giving each word its own special barcode.

In our list of words like man, woman, boy, girl, and so on, OHE turns each word into a string of numbers. The image you see shows this.

We've got nine words in our list, so each word gets turned into a line of nine numbers. For 'man', the first spot is a 1, and the rest are 0s. For 'woman', the second spot is a 1, and again, the rest are 0s. This pattern goes on for each word.

So, 'man' becomes 1 0 0 0 0 0 0 0 0, and 'woman' becomes 0 1 0 0 0 0 0 0 0. Each word has its own unique combination with one '1' and lots of '0s'. It's like every word has its own seat at a long table, and '1' means 'taken' while '0' means 'empty'. This helps the computer see each word as a different pattern of numbers, and that's the beginning of teaching a computer how to read.

When we use One-Hot Encoding (OHE), it's like we’re assigning each word a unique code that stands out in the crowd. It’s straightforward and it works, even with a lot of words.

Pros:

• Super Clear: Each word has its own unique code, so there’s no mix-up when the computer looks at them.
• Easy Peasy: It's simple to understand and to program, which is always nice.

Cons:

• Space Hog: If you have a ton of words, you end up with really long strings of numbers, which takes up a lot of computer memory.
• Misses the Connection: It doesn't really capture the relationship between words. 'King' and 'queen' might be close in meaning, but in OHE, they look totally different, just like 'king' and 'apple' would.

Use Cases:

• Simple Stuff: When you're working on basic tasks and you don’t have too many words to deal with, OHE is still a good go-to.
• Getting Started: It’s great for teaching beginners how to get machines to understand words.
• Quick Tests: If you want to test an idea quickly, OHE can help you set things up without much fuss.

Even today, OHE has its place in places where simplicity and clarity are more important than being fancy or super efficient.

It's important to note that while OHE provides distinct representations for each word, but modern embeddings go further by capturing semantic relationships, which OHE does not.

2. N-Grams - Cracking the Code of Context

After understanding the basics of One-Hot Encoding, let's delve into the concept of N-Grams, a smart way to catch the flow of a conversation or text.

An N-Gram is a chunk of 'N' consecutive words used to keep track of word sequences. The image shows three types: unigrams (single words), bigrams (pairs of words), and trigrams (triplets of words).

The Story of N-Grams:

N-Grams have been around for a while. They were first mentioned by Markov in the early 20th century and have been a staple in language processing ever since. These little snippets of language help computers predict what word might come next, based on the words that come before. It's like reading ahead to see what's around the language corner.

Pros:

• Better Than Solo: N-Grams understand the context better than single words alone.
• Pattern Spotting: They're great at spotting common phrases and predicting the next word in a series.

Cons:

• Memory Munchers: More N-Grams means more memory used, especially with bigrams and trigrams.
• Still a Bit Clueless: Even with the context, they can miss the bigger picture or the meaning of longer sentences.

Use Cases:

• Guessing Games: They help in tasks like auto-completing your search queries or predicting the next word as you type.
• Language Learning: N-Grams can help language models understand and generate language that sounds natural.

Even today, N-Grams are everywhere, from search engines that finish your sentences to chatbots that understand what you're getting at.

As for the original material, you might want to look up the works of Andrey Markov, who pioneered the study of sequences on which these N-Grams are based.

For the nitty-gritty details have a look at - https://www.decontextualize.com/teaching/rwet/n-grams-and-markov-chains/ .

N-Grams are fundamental to language models, but they're different from Markov models, even though both concepts deal with sequences. N-grams in language processing specifically look at word sequences without the probabilistic state transitions of Markov models.

3. Unlocking Language with Neural Networks: The 2003 Breakthrough

Back in 2003, a team led by Yoshua Bengio took a huge step forward in the world of NLP with the Neural Probabilistic Language Model. They showed us a new way to predict words and understand sentences using something called a neural network.

The Model Explained: The image you see is like a map of the neural network they designed. At the bottom, we have our words, each getting turned into a bunch of numbers by looking them up on a special table. This is where the word embeddings live. Then these numbers get mixed and mingled in a middle layer (that's the green web you're looking at) where they can learn from each other. The top layer decides the chances of what the next word could be, using a math trick called 'softmax'.

This was a pioneering step in using neural networks to learn dense representations of words in a continuous vector space, enabling the model to capture words in context

Pros:

• Smart Learning: It gets better as it goes, learning how words like to hang out together.
• Deep Thinking: This model can dig deeper into the meaning of words and sentences than simpler methods.

Cons:

• Hungry for Power: It needs a lot of computer juice to run and learn.
• Takes Its Time: Learning with this model isn't a quick thing; it's more of a slow cook.

For those who want to explore this model more, diving into Bengio's original paper "A Neural Probabilistic Language Model" will give you all the details.

This neural network was one of the first to get the ball rolling on understanding and predicting language in a way that goes beyond just counting words or looking at them one by one. It opened up a whole new avenue for machines to get what we're saying, and it's still influencing how we teach computers to process language today.

4. Revolutionizing Meaning Extraction: Collobert & Weston's 2008 Innovation

In 2008, the field of NLP was set to evolve once more with a contribution from Ronan Collobert and Jason Weston. They proposed a neural network architecture that was all about getting to the heart of what words mean, faster and more efficiently than ever before.

The Architecture Unveiled:

This new design was built for the big leagues—large-scale semantic extraction. It meant that not only could we teach computers to recognize words, but now they could also grasp the deeper meanings behind them at a pace and scale that was unheard of.

Pros:

• Deep Dive into Meaning: The architecture was a genius at understanding the complex web of word meanings.
• Speedy Processing: It made the whole process of embedding computations much faster.
• The architecture not only advanced word-level processing but also played a significant role in understanding the structure and meaning of sentences, marking a substantial advancement in NLP

Cons:

• Data Hungry: To really shine, it needed a lot of data to chew on.
• Resource Intensive: Despite its efficiency, the system required some serious computational muscle.

Collobert and Weston's 2008 architecture set a new standard in the efficiency of semantic understanding. It was a big leap towards computers not just reading words but getting them—a step closer to a world where machines can understand the subtleties of human language as more than just strings of letters.

5. Word2Vec's CBOW and Skip-Gram Models - Mapping Words with Precision:

In 2013, a team including Tomas Mikolov introduced Word2Vec, changing the landscape of word embeddings with two innovative architectures: Continuous Bag of Words (CBOW) and Skip-Gram.

The paper is explained here - https://arxiv.org/pdf/1411.2738.pdf

Word2Vec:

Word2Vec helped us see words in a new light. The CBOW model looks at the context - the words around a blank space - and then guesses the missing word. Skip-Gram flips this idea on its head; it starts with one word and then figures out the words that are likely to be around it.

Pros:

• Relationship Expert: Word2Vec is great at understanding how words are related to each other.
• Size Matters Not: It can handle big vocabularies without making the data too bulky.

Cons:

• Task Specific: CBOW is faster but a bit less precise, while Skip-Gram is slower but sharper, especially with rare words.

The introduction of Word2Vec marked a paradigm shift in how we develop language models, influencing a wide array of subsequent advancements in the field.

If you're up for some reading to get the full story, check out "Efficient Estimation of Word Representations in Vector Space " by Mikolov and his team. It's the paper where they first laid out these ideas.

The best explanation on Word2Vec is

6. Understanding Embedding Dimensions (V*D):

When we talk about V*D in embeddings, we're talking about the grid that holds all our word vectors. 'V' is how many words we've got, and 'D' is how many features each word has. It's like a big spreadsheet where each word has its own row of numbers that tells us all about it.

Word2Vec, especially with its CBOW and Skip-Gram models, was a significant leap forward in word embedding technology. It gave us tools that are better at capturing the subtleties of language and made it easier to work with large vocabularies without getting bogged down in data. These models remain essential in the world of NLP, from powering search engines to making smart assistants more helpful.