Introduction to Statistical NLP : Remembering Old Sports : Part - 1

In the realm of language and technology, where human expression meets computational power, a fascinating field has emerged: Statistical Natural Language Processing (NLP). This discipline bridges the gap between the intricacies of human language and the structured world of machines. By harnessing the power of statistics and probability theory, NLP algorithms delve into the vast corpus of text data, extracting meaning, understanding context, and even generating human-like text. From chatbots that engage in natural conversations to machine translation that breaks down language barriers, statistical NLP is revolutionizing the way we interact with machines and communicate with each other.

Bag-Of-Words and Term Frequency- Inverse Term Frequency are two important models in this realm.

In this article we will explore the gist of the Bag-Of-Words Model.

A bag of Words is a NLP Technique that represents text as a collection of words disregrading the grammar and word order.It is a simple but effective way to extract features from the text that can be used for a variety of NLP Tasks, such as text classification, machine translation and Sentiment Analysis.

To create a bag of words model, the text is first preprocessed by removing stop words (such as "the", "is","and","of") and then stemming or lemmatizing the words (to reduce them to their root form). The remaining words are then counted used to create a vector representation of the text. The vector representation is a list of numbers where each number corresponds to the count of a particular word in the text.

The scoring method we use here is to count the presence of each word and mark 0 for absence. This scoring method is used more generally.

A problem with scoring word frequency is that highly frequent words start to dominate in the document (e.g. larger score), but may not contain as much “informational content” to the model as rarer but perhaps domain specific words.

One approach is to rescale the frequency of words by how often they appear in all documents, so that the scores for frequent words like “the” that are also frequent across all documents are penalized.

This approach to scoring is called Term Frequency – Inverse Document Frequency, or TF-IDF for short

Bag Of Word smodels are often used in conjunction with ML algorithms to train text classifiers. For Example, BOW can be used to train a classifiers to distinguish between spam and legitimate emails.

The classifier could be trained on a dataset of labeled emails, where as each email is represented by a BOW. Once classifier is trained, it can be used to predict the label of new emails based on their bag of words representation.

Advantages Of Bag Of Words

1. Simple the understand and implement

2. Effective for a variety of NLP Tasks

3. Robust to noise and spelling Errors

Disadvantages Of Bag Of Words

1. Disregards grammar and word order

2. Can be sensitive to the choice Of Stop Words and Stemming/Lemmatization Algorithms.