Introduction to Natural Language Processing (NLP)

What is NLP?

NLP is the branch of Artificial Intelligence (AI) that helps computers understand, interpret, and respond to human language in a valuable way. Common applications include:

• Text Classification (e.g., spam detection, sentiment analysis)
• Machine Translation (e.g., translating English to French)
• Named Entity Recognition (NER) (e.g., finding names, dates, places in a sentence)
• Speech Recognition (e.g., converting spoken language into text)

Core Concepts in NLP

1. Tokenization: The process of breaking down text into smaller units (like words or phrases) to help machines process the data. For example: Sentence: “I love coding.” Tokens: [“I”, “love”, “coding”, “.”]
2. Stemming and Lemmatization: Simplifying words to their base forms. Stemming: “running” → “run” Lemmatization: “better” → “good”
3. Bag of Words (BoW): A method to represent text by counting how often each word appears in a sentence or document. It ignores grammar and word order but captures frequency.
4. TF-IDF (Term Frequency - Inverse Document Frequency): A scoring method to weigh the importance of words in a document relative to a corpus (collection of documents). Rare words in a large corpus get more weight.
5. Word Embeddings: A more advanced method where words are represented as continuous vectors in a high-dimensional space. Words with similar meanings are closer to each other in this space.

Getting Hands-on with NLP

To understand these concepts, you can try simple Python code using the Natural Language Toolkit (NLTK) and spaCy.

Installing Libraries:

!pip install nltk spacy        

Tokenization Example with NLTK:

import nltk
nltk.download('punkt')

# Tokenizing a sentence
from nltk.tokenize import word_tokenize
text = "Natural Language Processing is exciting!"
tokens = word_tokenize(text)
print(tokens)        

Tokenization with spaCy:

import spacy

# Load the spaCy model
nlp = spacy.load('en_core_web_sm')

# Process a sentence
doc = nlp("Natural Language Processing is exciting!")

# Tokenize and display each token
for token in doc:
    print(token.text)        

TF-IDF Example with scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer

# Example sentences
docs = ["I love coding", "coding is fun", "I love fun activities"]

# Create the TF-IDF model
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(docs)

# Display the TF-IDF matrix
print(tfidf_matrix.toarray())