Understanding machine learning and deep learning algorithms by comparing and contrasting

Background

I have a book by the painter Kandinsky. Its a great for understanding of his creativity.

Curiously, Kandinsky had a neurodiverse condition called? synesthesia. Synesthesia is a neurological phenomenon where stimulation of one sensory or cognitive pathway leads to involuntary experiences in a second sensory or cognitive pathway. People with synesthesia, might "see" sounds, "hear" colors, or associate specific letters or numbers with colors or textures.

Specifically, Kandinsky could “hear” colours.

This sensory intermixing probably accounts for the richness of his art and paintings.

Mixing ideas helps us to understand things more deeply and intuitively and in a creative manner.

I thought of applying this to understanding machine learning and deep learning algorithms by comparing and contrasting them i.e. mixing things that traditionally do not mix.

This is not easy - I am thankful to Amita Kapoor for her feedback to my ideas

Understanding Algorithm categorization

Before we mix, lets understand the categorization. Broadly, we can categorize algorithms as

Supervised Learning involves labeled data, with algorithms like Linear Regression, Decision Trees, and Neural Networks used to predict outcomes or classify data. Unsupervised Learning involves unlabeled data, focusing on finding structure in the data, using algorithms like K-Means Clustering and Principal Component Analysis (PCA).

Parametric Models (e.g., Linear Regression, Logistic Regression) assume a specific form for the function mapping inputs to outputs and have a fixed number of parameters. Non-Parametric Models (e.g., Decision Trees, KNN) do not assume a fixed form and can grow in complexity with more data.

Linear Models (e.g., Linear Regression, Logistic Regression) assume a linear relationship between features and outcomes. Nonlinear Models (e.g., Neural Networks, SVM with non-linear kernels) capture more complex relationships.

Generative Models (e.g., Naive Bayes, GANs) model the joint probability distribution? and can generate new data. Discriminative Models (e.g., Logistic Regression, SVM) model the conditional probability and focus on decision boundaries.

Instance-Based Learning (e.g., KNN) relies on storing instances and comparing new data points directly to them. Model-Based Learning (e.g., Decision Trees, Neural Networks) builds an abstract model based on the data.

Comparing and contrasting algorithms

1. Regression can be seen as a form of a classification and vice versa

The concepts of regression and classification are distinct but interconnected, and under certain circumstances, they can be reframed in terms of each other.?

Regression involves predicting continuous numerical values. However, you can interpret regression as a classification task by discretizing the continuous output into discrete classes.

Suppose you’re predicting house prices using a regression model.

If you group house prices into discrete categories, like:

$0–$100k → Class 0

$100k–$200k → Class 1

$200k–$300k → Class 2

Then, the task becomes a classification problem with each range treated as a class. A regression model can still perform this classification implicitly by outputting a continuous value and assigning it to the nearest class.

On the other hand, Classification can be seen as a Regression problem. Classification involves predicting discrete labels, but it can be reformulated as a regression problem by assigning numerical values to classes.

Suppose you are classifying images as either "cat" or "dog."

Assign:

Cat → 0

Dog → 1

Train a regression model to predict a value between 0 and 1.

During inference:

• Outputs close to 0 are interpreted as "cat."
• Outputs close to 1 are interpreted as "dog."

2. Tree-Based Models: Regression and Classification

Decision trees and random forests can handle both regression (predicting numerical outcomes) and classification tasks (categorical outcomes) with minimal adjustments to their structure. The splitting criteria differ: for regression, they minimize variance, while for classification, they maximize information gain or Gini impurity reduction.

3. Clustering as Precursor to Classification

Clustering algorithms like k-means can be used to group data into "pseudo-classes" as a precursor to supervised classification. Clustering is unsupervised and doesn't rely on labeled data, while classification strictly depends on labeled training datasets.

4. Neural Networks: Regression and Classification

Neural networks can be adapted to perform both regression (using linear activation functions in the output layer) and classification (using softmax or sigmoid activations). The loss functions differ significantly: regression uses Mean Squared Error (MSE) or Mean Absolute Error (MAE), while classification often uses Cross-Entropy loss.

5. Dimensionality Reduction as a Helper for Clustering and Classification

Techniques like PCA (Principal Component Analysis) are used to preprocess data for clustering and classification, reducing noise and computational cost. PCA focuses on preserving variance, while clustering focuses on grouping based on similarity, and classification focuses on learning boundaries.

6. Support Vector Machines (SVMs): Linear vs. Non-linear Decision Boundaries

SVMs can perform both regression (Support Vector Regression, SVR) and classification tasks by finding optimal margins. In classification, SVM seeks to maximize the margin between classes, while in regression, it minimizes the deviation within a defined margin.

7. Reinforcement Learning vs. Supervised Learning

Both can be used for decision-making tasks (e.g., controlling a robot or optimizing marketing strategies) and can involve neural networks as function approximators. Reinforcement learning relies on delayed rewards and learns from interaction with the environment, whereas supervised learning learns from immediate feedback from labeled data.

8. Neural Networks and K-Nearest Neighbors (KNN)

Despite being conceptually distinct, early layers of Neural Networks often extract features that make data more "KNN-friendly," enabling effective hybrid models.

9. Support Vector Machines (SVM) and Neural Networks

Both can represent decision boundaries, but SVM excels with smaller datasets and clear margins, while Neural Networks perform better with larger datasets.

10. Linear Regression and Neural Networks

A Neural Network with no hidden layers and a linear activation function behaves identically to Linear Regression.

11. Decision Trees and Gradient Boosting Machines (GBMs)

GBMs are an ensemble of weak learners, typically Decision Trees, showing how a simple algorithm can be extended for more complex scenarios.

12. Principal Component Analysis (PCA) and K-Means

PCA is often used as a preprocessing step for K-Means to reduce dimensions and improve clustering results.

13. K-Means and Gaussian Mixture Models (GMM)

GMM generalizes K-Means by incorporating probabilistic distributions, but they converge similarly when covariance is isotropic.

14. Naive Bayes and Logistic Regression

While both are classification algorithms, Naive Bayes is generative and works with probabilities, while Logistic Regression is discriminative.

15. Random Forests and Neural Networks Random Forests handle missing data better than Neural Networks, but the latter shines in learning intricate patterns with complete data.

16. Autoencoders and PCA

Autoencoders are nonlinear extensions of PCA, capable of capturing more complex features in lower-dimensional space.

I find this contrasting thinking better to understand algorithms.

About my work

We take a complex and holistic view of an AI Engineer in my new role as a senior AI fellow in the MOJ and UK government.

If you want to work with me, we are recruiting

If you want to study #AI with me at the #universityofoxford - please see my course on AI (almost full now - only last few places remaining).

If you want to be a part of my community see Creating a community (LinkedIn group) for my blog - where you can ask me questions re AI

Image source:?

?https://www.amazon.co.uk/Concerning-Spiritual-Dover-Fine-History/dp/0486234118/?