Understanding the fashion and chronology of algorithms

We don't think of algorithms in terms of fashionable trends .. but in some cases, its helpful to do so to gain a better understanding of how they are used in terms of solving problems.

The purpose of an algorithm is to solve a problem. In our case, for data science, the problem is data driven. The choice of the tool(algorithm) is made in terms of the problem at hand, the resources, constraints and also fashion(trend) - much on the lines of Maslow's quote "If the only tool you have is a hammer, you tend to see every problem as a nail.”

Over the last 50 years, machine learning and statistical algorithms have evolved significantly.

In the pre-1950s , we started with statistical methods. In the 1960s, we saw the introduction of Bayesian approaches for probabilistic inference to complement the frequentist approaches. However, from the standpoint of machine learning as we have seen before, both these approaches are statistical (because the underlying data distribution should be knowable).

The 1970s experienced an AI winter due to pessimism about machine learning effectiveness.

Slowly in the 1990s, we saw a shift to data driven approaches, with the popularity of support-vector machines (SVMs). SVMs helped to solve non linear problems through the kernel trick.

The 2000s saw increased use of kernel methods and unsupervised machine learning. The 2010s were defined by the feasibility of deep learning, making machine learning integral to many services and applications.

Finally, the 2020s have been characterized by generative AI, leading to revolutionary models like ChatGPT and Stable Diffusion, significantly impacting public consciousness and the commercial potential of AI.

Depending on who you ask, these perspectives may differ in emphasis, but the overall trend is accurate. Today, we are going towards more complex, non linear data driven problems which are driven by rich media. This is taking us away from traditional statistical roots because increasingly, we find that the distribution is unknowable in these problem statements.

Today, people tend to want to apply deep learning because its in fashion, Two decades ago, that maybe SVM .. two more decades ago - the dominant algorithm could be decision trees or logistic regression.

Understanding the dominant algorithm in an era helps us to decouple the trend effect

So, here are the main eras and their algorithms

1950s - 1960s

Linear Regression: Statistical modeling technique for predicting a linear relationship between variables.

1970s

Decision Trees: Introduced as a machine learning algorithm for decision-making.

Nearest Neighbors: The k-nearest neighbors algorithm for pattern recognition.

Generalized Linear Models (GLMs) by John Nelder and Robert Wedderburn in 1972, extending linear regression model to accommodate response variables that have error distribution models other than a normal distribution,. GLMs unify various types of regression models, including: Linear Regression for normally distributed responses; Logistic Regression for binary responses; Poisson Regression for count data.

1980s - 1990s

Support Vector Machines (SVM): Developed for classification and regression by Vapnik and Cortes.

Random Forest: Introduced as an ensemble learning method.

Naive Bayes: Probabilistic classifier based on Bayes' theorem.

Efron's Bootstrap Method: Statistical technique for estimating the sampling distribution of a statistic.

Hastie and Tibshirani's work on Generalized Additive Models (GAM): An extension of linear models to include non-linear components.

2000s - 2010s

Kernel Methods (e.g., Gaussian Processes): Non-parametric models for regression and classification.

Deep Learning Resurgence: Breakthroughs in training deep neural networks, leading to the popularity of architectures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

2010s - Present

XGBoost (Extreme Gradient Boosting): Powerful gradient boosting algorithm for classification and regression.

Deep Learning Dominance: Advancements in architectures (e.g., Transformer models), pre-training, and transfer learning.

For example, adaboost was introduced in 1990s. It laid the foundations of other boosting algorithms like xgboost, catboost, GBM(Gradient boosting machines) , lightGBM etc.

Sometimes, we see all these grouped together - and missing the chronology (and the prevailing trend) causes you to lose context.

In other words, you would (likely) not use adaboost in 2024, but would need to understand adaboost as a foundation to xgboost (which is likely what you would use)

image source: https://pixabay.com/photos/catwalk-models-women-fashion-1840941/