Embeddings - The Foundation

In the realm of cutting-edge language models, it's crucial not to overlook the foundational concepts amidst the excitement. Understanding the journey from individual words to BERT representations, along with the underlying motivations, is essential to unravel the mysteries of these models. Without this comprehension, they remain enigmatic black boxes, hindering our ability to harness and advance their capabilities. Mastering these fundamentals empowers us to build upon and utilize these models effectively, aligning with our desired goals. Let's embrace the importance of grasping the basics to unlock the true potential of large language models.

In this I am going to cover basics of embeddings which are intermediate elements that live within machine learning services to refine models and how they serve to various modern NLP techniques and approaches.

Representing text as numbers

Machine learning models can only understand numbers. So, when you want to feed text to a machine learning model, you need to convert the text into numbers first. This is called vectorization. There are multiple approaches to do this-

• One-hot encoding
• Encode each word as a unique number
• Word Embeddings

Please refer to https://www.tensorflow.org/text/guide/word_embeddings in case you want to learn simulation and about each of the above approach.

Embeddings are transformed data matrices used in deep learning to represent variables and capture their relationships in a condensed, multi-dimensional format. in easier words they are just numerical representation of text data in Vector or Tensor format.

Word embeddings, in simple terms, are numerical representations of words used in natural language processing tasks. They transform words into dense vectors or arrays of numbers, where each number captures different aspects of the word's meaning or context. These representations enable machines to understand and analyze the relationships between words, facilitating tasks such as language translation, sentiment analysis, and text generation. By encoding semantic and syntactic information, word embeddings help bridge the gap between human language and computational algorithms.

We often talk about item embeddings being in X dimensions, ranging anywhere from 100 to 1000, with diminishing returns in usefulness somewhere beyond 200-300 in the context of using them for machine learning problems. This means that each item (image, song, word, etc.) is represented by a vector of length X, where each value is a coordinate in an X-dimensional space.

Note - A tensor is nothing but multidimensional combination of Vector.

in next edition I am going to illustrate how it works in real world data and try to demystify. Stay tuned for more updates..

Thanks for reading!!