Week 6: Unsupervised Machine Learning: Practical Overview and Applications

In our previous article, we explored supervised learning in detail. This week, we will dive into another major branch of machine learning: unsupervised learning. We'll look at the definition and types of unsupervised learning, its key algorithms, and real-world applications.

This article aims to provide a clear overview and practical examples. It starts with the basics for non-technical readers and gradually moves into the logic and implementation of key algorithms, using simplified pseudo-code to explain how they work.

1- What is Unsupervised Learning?

It's a type of machine learning where the model is trained on an unlabeled dataset. Unlike supervised learning, where the goal is to predict a known output, unsupervised learning aims to discover hidden patterns or intrinsic structures in the input data.

Think of it as exploring a new city without a map or guide, discovering interesting places, and understanding the layout on your own. The model groups similar data points together and identifies patterns that help us make sense of the data.

1.1. Types of Unsupervised Learning:

1. Clustering: Involves grouping similar data points together. For example, grouping customers based on their purchasing behavior.
2. Dimensionality Reduction: Involves reducing the number of features in a dataset while retaining its essential information. For example, simplifying a dataset with many variables to make it easier to analyze and visualize.

1.2. Key Concepts in Unsupervised Learning:

1. Features: The input variables used to discover patterns. For example, in customer segmentation, features could include purchase history, age, and location.
2. No Labels: Unlike supervised learning, there are no target values or labels to predict.
3. Model: The mathematical representation that finds patterns in the data.
4. Clusters: Groups of similar data points identified by the model.
5. Principal Components: New features created by combining the original features in dimensionality reduction. This helps in simplifying the data while preserving important patterns.

2. Real-World Applications:

Unsupervised learning has numerous real-world applications across various domains. Here are some examples, emphasizing the use of features and pattern discovery:

2.1. Marketing:

• Market Basket Analysis: Identifying items frequently bought together (Features: Transaction data).
• Customer Segmentation: Grouping customers based on their browsing behavior (Features: Pages visited, time spent on site).
• Personalized Marketing: Creating targeted marketing campaigns based on customer segments (Features: Customer preferences, purchase data).

2.2. Healthcare:

• Genomics: Identifying patterns in genetic data to understand diseases (Features: Genetic markers, patient history).
• Patient Segmentation: Grouping patients based on medical history and symptoms (Features: Medical records, demographic data).

2.3. Finance:

• Anomaly Detection: Identifying fraudulent transactions (Features: Transaction amount, location, time).
• Portfolio Optimization: Reducing the dimensionality of financial data to identify key factors (Features: Asset prices, economic indicators).

2.4. Retail:

• Inventory Management: Grouping products based on sales patterns (Features: Sales data, seasonal trends).
• Service Usage Patterns: Identifying distinct usage patterns among customers to offer customized service plans (Features: Call durations, data usage, time of use).

2.5. Telecommunications:

• Network Optimization: Optimizing network performance by analyzing usage patterns (Features: Traffic data, service quality metrics).
• Churn Prediction: Identifying customers likely to leave the service (Features: Usage patterns, customer service interactions).

2.6. Manufacturing:

• Manufacturing Fault Detection: Identifying patterns in sensor data to predict equipment failures (Features: Sensor readings, operational data).
• Quality Control: Grouping similar production batches for analysis (Features: Production parameters, quality metrics).

2.7. Environmental Science

• Climate Pattern Analysis: Identifying patterns in climate data to understand weather trends (Features: Temperature, precipitation, wind speed).
• Wildlife Migration Tracking: Grouping animal movement patterns to study migration (Features: GPS tracking data, environmental factors).

2.8. Urban Planning

• Traffic Flow Analysis: Grouping traffic patterns to improve city planning (Features: Traffic sensor data, road usage statistics).
• Land Use Classification: Identifying different land use types based on satellite images (Features: Image pixel data, geographical information).

2.9. Social Networks:

• Community Detection: Identifying groups of users with similar interests (Features: User interactions, profiles).
• Influence Analysis: Finding key influencers in a network (Features: Connection patterns, activity data).

3. Let's Get More Technical

If you're curious about the technical details, this section is for you. We'll uncover more about unsupervised learning concepts, metrics, and key algorithms:

3.1. Feature Engineering and Data Preprocessing:

In unsupervised learning, as in supervised learning, feature engineering and data preprocessing are essential. However, the focus and techniques might differ to suit the goals of unsupervised learning, which primarily involves discovering patterns and structures in unlabeled data.

3.1.1. Feature Engineering

• Creating New Features: Just like in supervised learning, deriving new features from existing ones can help uncover hidden patterns. For instance, combining or transforming features to highlight underlying structures in the data.
• Dimensionality Reduction Techniques: Methods such as Principal Component Analysis (PCA) and t-SNE are crucial in unsupervised learning for reducing the number of features while retaining the essential information. This not only simplifies the dataset but also helps in visualizing high-dimensional data.

3.1.2. Feature Scaling

• Normalization: Rescaling features to a range of [0, 1] is often necessary to ensure that no single feature dominates the others due to its scale.
• Standardization: Rescaling features to have a mean of 0 and a standard deviation of 1 ensures that the features contribute equally to the distance calculations used in many clustering algorithms.

3.1.3. Data Cleaning

• Handling Missing Values: Similar to supervised learning, strategies include removing instances with missing values or imputing them with mean, median, or mode to maintain the dataset's integrity.
• Dealing with Outliers: Outliers can significantly impact unsupervised learning algorithms. Detecting outliers using statistical methods or visualization and handling them by either removing or transforming them is essential.

3.1.4. Data Transformation

• Log Transformation: Useful for skewed data to make it more normal distribution-like, which can improve the performance of clustering algorithms.
• Clustering Transformation: Techniques such as one-hot encoding can be used to transform categorical data into a numerical format suitable for clustering. This helps the algorithm understand categorical similarities and differences.

3.2. Evaluation Metrics:

Unlike supervised learning, unsupervised learning doesn't have clear-cut labels for evaluation. However, we can use some metrics to assess the quality of the models:

3.2.1. Clustering Metrics:

• Silhouette Score: Measures how similar an object is to its own cluster compared to other clusters.
• Davies-Bouldin Index: Evaluates the average similarity ratio of each cluster with its most similar cluster.
• Calinski-Harabasz Index: Assesses the variance ratio between clusters and within clusters.

3.2.2. Dimensionality Reduction Metrics:

• Explained Variance Ratio: Measures the proportion of data variance retained by each principal component.
• Reconstruction Error: Evaluates the difference between the original data and the reconstructed data from reduced dimensions.

3.3. Key Algorithms in Unsupervised Learning

Let's dive into the top 5 algorithms. We'll break down their concepts, share some practical examples, and explain how it works in simplified pseudo-code format to clarify the logic and steps of each algorithm:

3.3.1. K-Means Clustering

Concept:

K-Means is a popular clustering algorithm that partitions data into K distinct clusters based on feature similarity. Each cluster is represented by its centroid, which is the mean of all data points in the cluster.

Example:

Suppose you want to segment customers based on their purchasing behavior. K-Means can group customers into clusters where each cluster represents a group of customers with similar purchase patterns.

How It Works:

- Initialize K centroids randomly
- Repeat until convergence:
  - For each customer:
    - Calculate the distance to each centroid based on features such as purchase history and demographics
    - Assign the customer to the nearest centroid
  - Update centroids by calculating the mean of all customers in each cluster
- Return the final centroids and clusters        

3.3.2. Hierarchical Clustering

Concept:

Hierarchical clustering builds a tree-like structure of nested clusters by either merging or splitting clusters recursively. There are two main types: Agglomerative (bottom-up approach) and Divisive (top-down approach).

Example:

Hierarchical clustering can group genes with similar expression patterns in bioinformatics, creating a dendrogram to visualize the hierarchy of clusters.

How It Works (Agglomerative):

- Start with each gene as a single cluster
- Repeat until only one cluster remains:
  - Find the two closest clusters based on expression patterns
  - Merge them into a single cluster
- Return the hierarchy of clusters        

3.3.3. Principal Component Analysis (PCA)

Concept:

PCA is a dimensionality reduction technique that transforms high-dimensional data into a lower-dimensional form while retaining most of the variance in the data. It identifies the principal components, which are linear combinations of the original features.

Example:

PCA can reduce the dimensionality of a dataset with many features, such as a dataset of images with thousands of pixel values, making it easier to analyze and visualize.

How It Works:

- Standardize the data
- Calculate the covariance matrix
- Compute eigenvectors and eigenvalues
- Select the top k eigenvectors
- Transform the data using the selected eigenvectors
- Return the transformed data        

3.3.4. t-Distributed Stochastic Neighbor Embedding (t-SNE)

Concept:

t-SNE is a nonlinear dimensionality reduction technique that maps high-dimensional data to a lower-dimensional space, typically 2 or 3 dimensions, for visualization. It minimizes the divergence between probability distributions of data points in high-dimensional and low-dimensional spaces.

Example:

t-SNE is commonly used to visualize complex datasets like handwritten digits or word embeddings, where the structure of the data is difficult to capture in high dimensions.

How It Works:

- Compute pairwise similarities in high-dimensional space for each digit
- Define probability distributions for high and low-dimensional spaces
- Minimize divergence between distributions by adjusting point positions in low-dimensional space
- Return the low-dimensional representation        

3.3.5. Apriori Algorithm

Concept:

The Apriori algorithm is used for mining frequent itemsets and discovering association rules in transactional datasets. It identifies itemsets that appear frequently together and derives rules indicating how the presence of one item affects the presence of another.

Example:

In a retail setting, Apriori can identify products often bought together. For instance, parents buying baby products like diapers and formula also tend to buy more coffee. This helps in market basket analysis and cross-selling strategies.

How It Works:

1. Initialize candidate itemsets of length 1:
   - Start with each product as a single itemset.

2. Repeat until no more frequent itemsets are found:
   - Count the occurrences of each candidate itemset in the transaction dataset.
   - Retain the itemsets that meet the minimum support threshold.
   - Generate new candidate itemsets by joining the retained itemsets.

3. Generate association rules from the frequent itemsets:
   - For each frequent itemset, find all non-empty subsets.
   - For every subset, calculate the confidence of the rule: (itemset - subset) => subset.
   - Retain the rules that meet the minimum confidence threshold.

4. Return the association rules:
   - Rules like "If diapers, then coffee" can be derived if they meet the support and confidence thresholds.        

3.4. Other Common Algorithms

Here's a brief explanation of other significant algorithms:

• DBSCAN: Density-based clustering algorithm that can find arbitrarily shaped clusters and detect noise. Example: Grouping customers with varying spending patterns.
• Isolation Forest: Anomaly detection algorithm that isolates observations by randomly selecting features and splitting values. Example: Detecting fraudulent credit card transactions.
• Latent Dirichlet Allocation (LDA): Topic modeling algorithm for discovering abstract topics in a collection of documents. Example: Identifying topics in a set of news articles.
• Autoencoders: Neural networks used for unsupervised learning tasks like dimensionality reduction and anomaly detection. Example: Reducing the dimensions of high-resolution images while preserving important features.

3.5. Common Challenges and Methods

Here are some common challenges in unsupervised learning and the methods used to overcome them.

Dealing with High-Dimensional Data

• Curse of Dimensionality: The difficulty of clustering and analyzing data in high-dimensional spaces.
• Techniques to Address It: Using dimensionality reduction techniques like PCA and t-SNE to simplify the dataset.

Model Selection and Validation in Unsupervised Learning

• Cluster Validation Techniques: Methods like silhouette score and Davies-Bouldin index to evaluate the quality of clustering.
• Choosing the Right Number of Clusters: Techniques such as the Elbow method and silhouette analysis to determine the optimal number of clusters.

Handling Imbalanced Data in Clustering

• Challenges: Dealing with clusters of varying sizes and densities.
• Techniques to Address Them: Using algorithms like DBSCAN that can handle varying cluster densities and sizes.

Are you a developer interested in practical examples?

The practical exercises in the excellent notebook below will help you solidify key concepts in unsupervised learning. It dives into techniques like clustering with K-Means, teaching you how to apply it, visualize decision boundaries, handle variability, and determine the best number of clusters. It also covers DBSCAN, spectral clustering, agglomerative clustering, and Gaussian mixtures for both clustering and anomaly detection.

The project is created by Aurélien Géron the Author of the book "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow" and the Former PM of YouTube video classification.

https://colab.research.google.com/github/ageron/handson-ml3/blob/main/09_unsupervised_learning.ipynb

Conclusion

Unsupervised learning is a major branch of machine learning that focuses on discovering hidden patterns and structures in data without labeled outputs. We explored its key concepts, such as clustering and dimensionality reduction, and discussed real-world applications across various domains. We also explored the technical details by reviewing essential algorithms like K-Means, hierarchical clustering, PCA, t-SNE, and Apriori, and listed other common algorithms.

Learning these core techniques will empower you to tackle a wide range of challenges in unsupervised learning and enhance your ability to extract meaningful insights from unlabeled data.

In this Zero to Hero: Learn AI Newsletter, we will publish one article weekly (or biweekly for in-depth articles). Next week, we'll dive deeper into Reinforcement Learning. Check out the plan of this series here:

AI Learning Paths: What to Learn and What's the Plan?

Share your thoughts, questions, and suggestions in the comments section.

Help others by sharing this article and join us in shaping this learning journey ????.