Week 5: Supervised Machine Learning: A Simplified In-Depth Explanation

In our previous article, we introduced supervised learning briefly. Today, we will dive deeper into this major branch of machine learning. We'll explore the definition and types of supervised learning, its key algorithms, and real-world applications.

This article provides a clear overview and practical insights. It starts with the basics for non-technical readers and gradually moves into the logic and implementation of key algorithms, using simplified pseudo-code to explain how they work.

1. What is Supervised Learning?

Supervised learning is a type of machine learning where the model is trained on a labeled dataset to learn the relationship between inputs and outputs.

Think of it as finding a pattern or rule that links your information (input features) to the answers you want (target labels). The model can make accurate guesses about new data that we haven't seen before. This is like using what humans know from past experiences to make predictions about the future.

As we will explore in this article, supervised learning is popular across many industries due to its ability to support informed decision-making.

1.1. Types of Supervised Learning: Regression vs. Classification

1. Regression: Involves predicting a continuous output variable. For example, predicting the price of a house based on its features.
2. Classification: Involves predicting a categorical output variable. For example, determining whether an email is spam or not.

1.2. Key Concepts in Supervised Learning

1. Features: The input variables used to make predictions. For example, in a house price prediction model, features could include square footage, number of bedrooms, and location.
2. Labels: The output variables or target values. For example, the actual prices of houses in the house price prediction model.
3. Training Set: The portion of the data used to train the model.
4. Test Set: The portion of the data used to evaluate the model.
5. Model: The mathematical representation that maps inputs to outputs.
6. Prediction: The output produced by the model for new input data.

2. Applications in Real-World Scenarios

The best way to understand when and how to use supervised learning is by looking at practical examples. Supervised learning has numerous real-world applications across various domains. Here are some examples, emphasizing the use of features and labels:

1. Healthcare

• Disease Diagnosis: Predicting whether a tumor is malignant or benign (Features: Medical scan attributes; Labels: Malignant or benign).
• Predictive Analytics: Predicting patient outcomes and risk factors (Features: Medical history, lab results, demographic information; Labels: Patient outcomes).

2. Finance

• Stock Price Prediction: Predicting future stock prices (Features: Historical prices, trading volumes, economic indicators; Labels: Future stock prices).
• Credit Scoring: Assessing creditworthiness (Features: Loan application details, repayment histories; Labels: Credit score or default status).

3. Marketing

• Customer Segmentation: Classifying customers into segments (Features: Purchasing behavior, demographics, browsing history; Labels: Customer segments).
• Targeted Advertising: Predicting ad clicks (Features: User activity, ad characteristics; Labels: Click or no click).

4. Transportation

• Traffic Prediction: Predicting traffic patterns and congestion (Features: Historical traffic data, weather conditions, events; Labels: Traffic levels).
• Route Optimization: Finding the most efficient routes (Features: Past delivery times, traffic data, road conditions; Labels: Optimal routes).

5. Retail

• Inventory Management: Predicting product demand (Features: Historical sales data, seasonal trends, promotions; Labels: Future product demand).
• Personalized Recommendations: Suggesting products (Features: Past purchases, browsing behavior; Labels: Recommended products).

6. Education

• Student Performance Prediction: Identifying at-risk students (Features: Academic records, attendance, participation; Labels: Performance outcomes).
• Personalized Learning: Creating customized learning plans (Features: Learning styles, past performance; Labels: Recommended learning paths).

7. Agriculture

• Crop Yield Prediction: Predicting yields (Features: Soil quality, weather conditions, farming practices; Labels: Crop yields).
• Disease Detection: Identifying crop diseases (Features: Images of plants; Labels: Disease presence or absence).

8. Manufacturing

• Quality Control: Detecting product defects (Features: Images and sensor data from production; Labels: Defect or no defect).
• Predictive Maintenance: Predicting machine failures (Features: Historical maintenance records, sensor data; Labels: Failure or no failure).

9. Telecommunications

• Churn Prediction: Predicting customer churn (Features: Usage patterns, service call data, contract details; Labels: Churn or retain).
• Network Optimization: Optimizing network performance (Features: Historical traffic loads, network configurations; Labels: Traffic load predictions).

10. Human Resources

• Employee Attrition: Predicting employee turnover (Features: Job satisfaction, performance reviews, salary data; Labels: Stay or leave).
• Recruitment: Screening job applicants (Features: Resume details, past experiences; Labels: Suitable or not suitable).

3. Let's Get More Technical

If you're curious about the technical details and interested in a deeper understanding, this section is for you. We'll uncover more about supervised learning concepts, metrics, and key algorithms:

3.1. Feature Engineering and Data Preprocessing

Feature Engineering

• Creating New Features: Deriving new features from existing ones to improve model performance.
• Feature Selection Techniques: Methods like forward selection, backward elimination, and recursive feature elimination to select the most important features.

Feature Scaling

• Normalization: Rescaling features to a range of [0, 1].
• Standardization: Rescaling features to have a mean of 0 and a standard deviation of 1.

Data Cleaning

• Handling Missing Values: Strategies include removing instances with missing values or imputing them with mean, median, or mode.
• Dealing with Outliers: Outliers can be detected using statistical methods or visualization and then handled by either removing or transforming them.

3.2. Evaluation Metrics:

Classification Metrics

• Accuracy: The ratio of correctly predicted instances to the total instances.
• Precision: The ratio of correctly predicted positive observations to the total predicted positives.
• Recall: The ratio of correctly predicted positive observations to all observations in the actual class.
• F1-Score: The weighted average of Precision and Recall.
• ROC-AUC: A measure of the model's ability to distinguish between classes.

Regression Metrics

• Mean Absolute Error (MAE): The average of the absolute errors between predicted and actual values.
• Mean Squared Error (MSE): The average of the squared errors between predicted and actual values.
• R-squared: The proportion of the variance in the dependent variable that is predictable from the independent variables.

3.3. Key Algorithms in Supervised Learning

In this section, we explore the top 5 algorithms in detail, explaining their concepts, providing practical examples, and presenting how they work with pseudo-code for developers who prefer a structured implementation guide.

3.3.1. Linear Regression

Concept:

Linear regression models the relationship between a dependent variable (what you want to predict) and one or more independent variables (features) using a linear equation.

It helps us predict a target value (like house prices) based on other related features (like square footage, number of bedrooms, and location). It does this by finding the best-fitting straight line through the data points.

Example:

Suppose you want to predict the price of a house. You have data on other houses, including their prices and features like size, number of bedrooms, and location. By finding the best-fitting line through this data, you can predict the price of a new house based on its features.

How It Works:

I. Initialize Weights and Bias: We start with an initial guess for the line's slope (weights) and where it crosses the y-axis (bias).

II. Training Process: For each example in our data (each house's features and price):

1. Predict the Price: Use the current line to predict the price
2. .Calculate the Error: Find out how far off the prediction is from the actual price.
3. Adjust the Line: Make small adjustments to the weights and bias to reduce the error.
4. Repeat this process for many iterations (epochs) to improve the line's fit.

III. Final Model: After many adjustments, we get a line that predicts prices well based on the given features.

Pseudo-Code:

1. Initialize weights (w) and bias (b)

2. For each iteration (epoch):
   a. For each training example (x, y):
      i. Predict the output: y_pred = w * x + b
      ii. Calculate the error: error = y_pred - y
      iii. Update weights: w = w - learning_rate * error * x
      iv. Update bias: b = b - learning_rate * error

3. Return the weights and bias        

3.3.2. Logistic Regression

Concept:

Logistic regression is used for binary classification, meaning it helps us decide between two classes (like yes or no, spam or not spam). It predicts the probability that a given input belongs to a certain class. Unlike linear regression, which predicts a continuous value, logistic regression predicts a probability between 0 and 1.

Example:

Logistic regression can help determine if an email is spam or not based on features like the presence of certain keywords, frequency of specific terms, and other characteristics. By analyzing these features, the model predicts the probability that an email is spam and classifies it accordingly.

How It Works:

I. Initialize Weights and Bias: We start with an initial guess for the weights (slopes) and bias (intercept).

II. Training Process: For each example in our data (e.g., each email and its spam or not spam label):

• Calculate the Linear Combination: Combine the email features (like the presence of certain keywords) using the weights and add the bias.
• Apply the Sigmoid Function: Convert the linear combination into a probability using the sigmoid function, which outputs a value between 0 and 1, representing the likelihood of the email being spam.
• Calculate the Error: Find out how far off the prediction is from the actual label (spam or not spam).
• Adjust the Weights: Make small adjustments to the weights to reduce the error.
• Adjust the Bias: Make small adjustments to the bias to reduce the error.

III. Repeat this process for many iterations (epochs) to improve the model's accuracy.

Pseudo-Code:

1. Initialize weights (w) and bias (b)

2. For each epoch:
   a. For each training example (x, y):
      i. Calculate the linear combination: z = w * x + b
      ii. Apply the sigmoid function: y_pred = 1 / (1 + exp(-z))
      iii. Calculate the error: error = y_pred - y
      iv. Update weights: w = w - learning_rate * error * x
      v. Update bias: b = b - learning_rate * error

3. Return the weights and bias        

3.3.3. Decision Trees

Concept:

Decision trees are used to make predictions by recursively splitting the training data into subsets based on the feature that provides the best split. The goal is to achieve the highest information gain or the lowest Gini impurity at each split. Think of it like a flowchart where each decision leads to further branching until a final decision is made.

Gini impurity measures how mixed the classes are in a subset of data. A Gini impurity of 0 means all elements are of the same class, while a higher value means a more mixed set.

Use Case:

Decision trees can help predict whether a customer will leave a service (churn) based on various features such as usage patterns, customer demographics, and service history. By analyzing these features, the model creates a series of decision rules to classify customers as likely to churn or not.

How It Works:

1. Check for Base Case: If all data points in a subset belong to the same class (e.g., all customers either churn or not churn) or a maximum depth is reached, create a leaf node with the majority class or average value.
2. Find the Best Split: Identify the feature and threshold that best split the data. This might be based on criteria like highest information gain or lowest Gini impurity. For example, you might find that usage patterns best predict whether a customer will churn.
3. Split the Data: Divide the data into two subsets based on the chosen feature and threshold (e.g., customers who use the service more than a certain number of hours vs. those who use it less).
4. Create a Decision Node: Form a decision node that represents the chosen feature and threshold.
5. Recursively Split: Repeat the process (steps 1-4) for each subset. This means creating further splits within the groups of customers based on other features like demographics or service history.
6. Return the Decision Node: Once the process is complete, the decision node (tree) is returned, which can be used to make predictions.

Pseudo-Code:

1. If all data points in a subset belong to the same class (all customers churn or don't churn) or the maximum depth is reached:
   - Return a leaf node with the majority class (e.g., "churn" or "not churn").

2. Find the feature (e.g., usage pattern) and threshold (e.g., hours > 5) that best split the data.

3. Split the data into two subsets:
   - Subset 1: Customers who use the service more than 5 hours.
   - Subset 2: Customers who use the service 5 hours or less.

4. Create a decision node with the chosen feature and threshold:
   - For example, "If usage pattern > 5 hours, go to Subset 1; else go to Subset 2."

5. Recursively repeat steps 1-4 for the left (Subset 1) and right (Subset 2) subsets:
   - For Subset 1: Check if further splits based on other features like demographics or service history improve the classification.
   - For Subset 2: Similarly, check for further splits.

6. Return the decision node (tree).        

3.3.4. Support Vector Machines (SVM)

Concept:

Support Vector Machines (SVM) are used for classification tasks. They work by finding the hyperplane (a line in 2D, a plane in 3D, or a higher-dimensional surface) that best separates the data into different classes. The goal is to maximize the margin, which is the distance between the hyperplane and the nearest data points of each class (these points are called support vectors).

Example:

SVMs can classify images as cats or dogs based on pixel values. By finding the best hyperplane that separates images of cats from images of dogs, the model can accurately classify new images.

How It Works

I. Initialize Weights and Bias: Start with initial guesses for the weights (slopes) and bias (intercept).

II. Training Process:

For each example in our data (e.g., each image and its label, cat or dog):

• Calculate the Decision Function: Combine the input features (pixel values) using the weights and add the bias.
• Check the Margin: If the product of the label and the decision function is less than 1 (meaning the example is within the margin or misclassified): Update Weights: Adjust the weights to move the example closer to the correct side of the hyperplane. Update Bias: Adjust the bias to help move the example to the correct side.
• Regularize Weights: If the example is correctly classified and outside the margin, adjust the weights slightly to maintain the margin and prevent overfitting.

III. Final Model: After many adjustments, the model finds the optimal hyperplane that best separates the classes.

Pseudo-Code:

1. Initialize weights (w) and bias (b)

2. For each epoch:
   a. For each training example (x, y):
      i. Calculate the decision function: z = w * x + b
      ii. If y * z < 1:
         - Update weights: w = w + learning_rate * (y * x - 2 * regularization_strength * w)
         - Update bias: b = b + learning_rate * y
      iii. Else:
         - Update weights: w = w - learning_rate * 2 * regularization_strength * w

3. Return the weights and bias        

3.3.5. k-Nearest Neighbors (k-NN)

Concept:

k-Nearest Neighbors (k-NN) is a simple and intuitive algorithm used for classification and regression tasks. It predicts the target value for a new example based on the majority class (for classification) or average value (for regression) of the k-nearest neighbors in the feature space.

Example:

k-NN can recommend products to customers based on the preferences of similar customers. By finding the k most similar customers (neighbors) and looking at their preferences, the algorithm can suggest products that are likely to be of interest to the new customer.

How it Works:

I. Calculate Distances: For each new example (e.g., a customer looking for product recommendations), calculate the distance between the new example and all the training examples (e.g., other customers with known preferences).

II. Select Neighbors: Sort the distances and select the k-nearest neighbors (e.g., the k most similar customers).

III. Make Prediction:

• For Classification: Return the majority class among the k-nearest neighbors (e.g., if most similar customers bought a specific product, recommend that product).
• For Regression: Return the average value of the target variable among the k-nearest neighbors (e.g., predict the average spending based on similar customers).

Pseudo-Code:

1. For each new customer:
   a. Calculate the distance between the new customer and all existing customers based on their preferences.
      - Use a distance metric like Euclidean distance.
   b. Sort the distances to find the k-nearest neighbors.
      - Select the top k closest customers.
   c. For classification: Recommend the product most frequently bought by the k-nearest neighbors.
      - Count the frequency of each product in the k-nearest neighbors.
      - Return the product with the highest count.
   d. For regression: Predict the average spending of the k-nearest neighbors.
      - Sum the spending values of the k-nearest neighbors.
      - Return the average value.
        

Other Common Algorithms:

Here's a brief introduction to other significant algorithms:

• Naive Bayes is a probabilistic classifier based on Bayes' theorem, assuming independence between features. Top use cases include text classification, such as classifying news articles into categories, and spam detection.
• Random Forests is an ensemble method that combines multiple decision trees to improve accuracy and control over-fitting. Top use cases include predicting loan defaults and disease diagnosis.
• Gradient Boosting Machines (GBM) is an ensemble technique that builds trees sequentially, each new tree correcting errors from the previous ones. Top use cases include fraud detection and predicting sales.
• AdaBoost is an ensemble method that combines weak classifiers into a strong one by focusing on errors. Top use cases include image classification and face detection.
• Neural Networks are computational models inspired by the human brain, capable of learning complex patterns from data. Top use cases include image recognition and speech recognition.
• XGBoost is an optimized gradient boosting algorithm designed for speed and performance. Top use cases include predictive modeling in competitions and stock price prediction.
• LightGBM is a highly efficient gradient boosting framework that uses tree-based learning algorithms. Top use cases include large-scale data analysis and ranking problems in search engines.

Advanced Topics

Overfitting and Underfitting

• Overfitting: When a model performs well on training data but poorly on new data.
• Underfitting: When a model performs poorly on both training and new data.
• Techniques to Address Them: Use of regularization, cross-validation, and simpler models.

Model Selection and Hyperparameter Tuning

• Grid Search and Random Search: Techniques for finding the best hyperparameters.
• Automated Tools (AutoML): Tools that automate the process of model selection and hyperparameter tuning.

Ensemble Methods

• Bagging: Combining the predictions of multiple models to reduce variance.
• Boosting: Combining the predictions of multiple models to reduce bias.
• Stacking: Combining multiple models' predictions using another model.

Are you interested in a practical end-to-end example? Check out this notebook project that predicts median house values in California districts using various features. the project is created by Aurélien Géron the Author of the book "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow" and the Former PM of YouTube video classification.

https://colab.research.google.com/github/ageron/handson-ml3/blob/master/02_end_to_end_machine_learning_project.ipynb

Conclusion

Supervised learning is a major branch of machine learning that involves training models on labeled datasets to predict outcomes for new, unseen data. We explored its key concepts, such as features, labels, and types like regression and classification. We also looked at real-world applications across various domains and then dove into the technical details by reviewing the top 5 essential algorithms, including linear regression, logistic regression, decision trees, SVM, and k-NN, and listed other common algorithms. By understanding these foundational elements, you can effectively recognize the proper use cases of supervised learning and gain a solid starting point for applying them in your projects.

In this Zero to Hero: Learn AI Newsletter, we will publish one article weekly (or biweekly for in-depth articles). Next week, we'll dive deeper into Unsupervised Machine Learning. Check out the plan of this series here:

AI Learning Paths: What to Learn and What's the Plan?

Share your thoughts, questions, and suggestions in the comments section. Help others by sharing this article and join us in shaping this learning journey ????.