Breaking Down Machine Learning Algorithms: A Beginner’s Guide to Linear Regression

Introduction to Supervised Learning

Before diving into Linear regression, it's important to understand its broader category: Supervised learning. Linear regression is one of the key methods within supervised learning.

In supervised learning, an algorithm learns from a labeled dataset where both the input features (independent variables) and their corresponding outputs (dependent variables) are provided. The goal of the algorithm is to map inputs to outputs so it can accurately predict outcomes for new, unseen data.

Mileage (km) Selling Price ($)

24000 km 25,000

32000 km 22,000

40000 km 19,000

This dataset illustrates how the distance driven (mileage) affects the selling price of used cars. After identifying the pattern or relationship between the mileage (input) and selling price (output), we can use this information to predict the price of newly input data. Supervised learning is like a student being taught with the answers, so they can learn the patterns and solve similar problems later on.

Linear Regression Overview

Linear regression is a supervised machine learning algorithm used to predict a number (continuous outcome - dependent variable) based on one or more input factors (features - independent variable). It’s called "linear" because it assumes a straight-line relationship between the input and the output. It's mainly used in data science for tasks like predicting sales, pricing and forecasting.

Types of Linear Regression:

• Simple Linear Regression: Involves only one independent variable. For example, predicting house prices based on square footage.
• Multiple Linear Regression: Involves two or more independent variables. For example, predicting house prices based on square footage, number of bedrooms, and location.

In this guide , we will use linear regression to predict price of a car based on its various features such as mileage, age, engine size, horsepower.

Mathematical Explanation of Linear Regression

General Equation: Linear regression models the relationship between input and an outcome/output using the equation of a straight line .

The formula for linear regression can be expressed as:

y = β0 + β1x1 + β2x2 + ... + βnxn + ?

• y = Predicted value (e.g., selling price of a car)
• x1,x2, ...... = Independent variables (e.g., mileage, age)
• β0 = Intercept (the predicted value when all independent variables are zero)
• β1,β2,...= Coefficients for each independent variable (how much each variable contributes to the prediction)
• ? = Error term (the difference between predicted and actual values)

The coefficients β1,β2,... represent the expected change in the predicted outcome (e.g., selling price) for a one-unit change in each independent variable (e.g., mileage, age), holding all other variables constant. This means that as the independent variables change, the predicted selling price will change in proportion to their respective coefficients.

Car Price Prediction Example :

For our car price prediction the equation looks like this:

selling price = β0 + β1?Mileage + β2?Age + β3?Engine?Size + β4?Horsepower + ?

"For every Horsepower increases, the price might increase by a certain amount, depending on the value of β4"

This General equation can also be writern as matrix form,

y = X ?β + ?

• y is the vector of target variable (selling price)
• X is the matrix of features variables ( i.e., x1,x2, ....Independent variables (e.g., mileage, age))
• β vector of coefficients
• ? is the vector of Error term.

Optimizing Linear Regression: How to Find the Best Line?

After setting up the linear regression equation, the next goal is to find the best-fit line that minimizes the difference between actual and predicted values. This is done by determining the optimal values for the coefficients (β’s) in the equation.

Loss Function

In linear regression, the line that fits the data best is determined by minimizing the loss function, which measures how far the predicted values are from the actual ones. The most commonly used loss function in linear regression is the Mean Squared Error (MSE):

MSE :

Mean Squared Error (MSE) is one of the most commonly used loss functions in regression problems, particularly in linear regression. It measures the average of the squared differences between the actual values and the predicted values. The idea is to quantify how well the model's predictions match the actual data points.

MSE = (1/n) ∑ ( yi?y_hat)^2

• n is the number of data points (the total number of observations).
• yi represents the actual value of the dependent variable (the true value).
• y_hat represents the predicted value from the model.

eg :

Mileage(km) Actual Selling Price($) Predicted Price ($) Residual (Error) Squared Error

24,000 25,000 24,000 1,000 1,000,000

32,000 22,000 21,500 500 250,000

40,000 19,000 18,500 500 250,000

The MSE would be the average of the squared errors (1,000,000, 250,000, 250,000):

MSE = (1,000,000+250,000+250,000) / 3=500,000

How Do We Minimize This Loss?

To reduce the MSE, we adjust the coefficients (β's) in our linear equation. This process is called optimization, and one popular optimization technique is Gradient Descent.

The scatter plot displays a set of data points representing the relationship between mileage (the number of kilometers driven) and selling price (in dollars) for used cars. Each blue dot represents a specific car, where the horizontal axis shows its mileage and the vertical axis shows its corresponding selling price.

The goal of linear regression is to find a straight line that best fits this data, meaning it accurately predicts the selling price of a car based on its mileage. Like the green line.

The red dashed lines are the starting guesses for the regression model. These lines represent different possible straight lines (or models) that try to predict the selling price based on the mileage.

At this point:

• The lines have different slopes (how steep the line is), which means they predict different selling prices for the same mileage.
• They also all start from a common point, often the origin (0, 0), which assumes that when the mileage is zero, the selling price is zero. This is an assumption that the model will later adjust if it's not true.

The green line is the final outcome of this process—it is the best-fit line that minimizes the prediction error, which means it passes as close to the blue data points as possible. This line is the result of optimizing the slope and intercept so that the predicted selling price (from the line) is as accurate as possible for any given mileage.

To find the optimal values of β, we need to minimize the difference between the actual values and the predicted values by reducing loss function.

Gradient Descent: Finding the Optimal Coefficients (β's)

Gradient Descent is an iterative optimization algorithm that adjusts the values of the coefficients to minimize the MSE. Here's how it works:

1. Start with initial guesses for the coefficients (which could be random).
2. Calculate the error by using the current coefficients and computing the MSE.
3. Update the coefficients: Based on the direction and magnitude of the error, adjust the coefficients step by step to reduce the error.
4. Repeat the process until the error converges, meaning it cannot be reduced any further.

The core idea is to iteratively adjust the coefficients to move "down" the slope of the error curve until the MSE reaches its lowest point. This is why it’s called gradient "descent" — we’re descending down a slope of errors.

Where:

• βj is the jth coefficient (this could be β0 for the intercept, or β1,β2,… for the other features).
• α is the learning rate, which controls the size of the steps taken towards the minimum of the cost function.
• J(β) is the cost function (typically, the Mean Squared Error in linear regression).
• ?(J(β)/?βj is the partial derivative of the cost function with respect to βj, which indicates how the cost function changes as βj changes.

Where:

• n is the number of data points.
• yi is the actual value of the target variable.
• y_hat is the predicted value based on the current coefficients.
• xij is the jth feature (independent variable) for the ith data point.

The image illustrates the process of gradient descent, an optimization technique used to minimize the cost function J(w), which measures how far the model's predictions are from the actual values. The vertical axis represents the value of this cost function, while the horizontal axis represents the model's parameters, specifically the weights or coefficients w. The gradient descent algorithm starts at an initial weight, where the goal is to iteratively adjust the weights to minimize the cost function. At the initial weight, the gradient (depicted by the dashed line) represents the slope of the cost function, indicating how steep the curve is and which direction the weights need to move to reduce the cost. The arrows in the image illustrate this movement, showing how the algorithm moves in the opposite direction of the gradient to minimize the cost. Through repeated steps, the gradient descent algorithm continues to update the weights until it reaches the global cost minimum, Jmin(w), where the slope of the cost function becomes zero or nearly zero, signifying the lowest possible cost value.

Practical Applications of Linear Regression

Linear regression is widely used in various industries to solve practical problems. Here are some common business use cases where it plays a crucial role:

1. Sales Forecasting

In retail and e-commerce, predicting future sales is essential for making informed decisions about inventory, marketing, and staffing. Linear regression helps businesses forecast sales based on past data, such as monthly sales figures, seasonal trends, and promotional campaigns. By identifying patterns in these factors, companies can predict future sales and adjust their strategies accordingly.

Example: A clothing retailer might use linear regression to predict next month’s sales by analyzing historical data like past sales, holidays, and discount periods. If the model identifies a strong relationship between sales and promotional discounts, the company can plan future discounts to boost sales during slower periods.

2. Inventory Management

Inventory optimization is critical to prevent both overstocking and stockouts. Linear regression helps businesses predict product demand based on factors such as previous sales, time of year, and market trends. By accurately forecasting demand, companies can maintain the right amount of stock and reduce storage costs while meeting customer needs.

Example: A supermarket can use linear regression to forecast demand for certain products during the holiday season, ensuring that they have enough stock of high-demand items like seasonal foods or gifts, without over-ordering products that won’t sell as quickly.

3. Market Analysis

Linear regression can also help businesses understand the relationship between various market factors and customer behavior. For instance, it can be used to analyze how pricing, advertising spend, or economic factors affect consumer purchasing decisions.

Example: A company might use linear regression to study how their advertising budget influences revenue. By modeling the relationship between ad spend and sales, they can determine the optimal budget allocation for future campaigns to maximize their return on investment (ROI).

Sample Python code