From Loss to Learning: The Importance of Cost Functions in Machine Learning

Understanding the fundamentals of machine learning often begins with mastering the concept of cost functions. In the context of linear regression, the cost function is a crucial component that helps in evaluating the performance of a model. Let's break down this concept with an intuitive example.

What is a Cost Function?

A cost function measures how well a machine learning model is performing. Specifically, it quantifies the error between the predicted values by the model and the actual target values from the training data. For linear regression, we use a function to fit a line to the training data. The goal is to minimize the error between the predicted values and the actual values, thereby finding the best-fit line.

Linear Regression Model

In linear regression, we have a model represented by the function f{w,b}(x) = wx + b, where w (weight) and b(bias) are the parameters of the model. These parameters determine the slope and the intercept of the line, respectively. By adjusting w and b, we can find the line that best fits the data.

For example:

- If w = 0 and b = 1.5 , the line is horizontal at y = 1.5 .

- If w = 0.5 and b = 0, the line has a slope of 0.5 and passes through the origin.

Measuring the Fit: The Cost Function

To determine how well a line fits the data, we use a cost function. One commonly used cost function for linear regression is the Mean Squared Error Cost Function:

Squaring this error ensures that both positive and negative differences contribute equally to the cost.

• y-hat (the y with a little symbol over it) is a variable used in statistics to represent the predicted value of our model when training.
• y is the variable that represents the actual value provided in the training dataset.
• The i subscript below y and y-hat signifies the i’th point in the dataset.
• m is the number of data points in our dataset.
• J(b_0, b_1) is the cost, which we will discuss now.

The purpose of the Mean Squared Error Cost Function is to minimize this error as much as possible for all the points in the dataset.?

Visualizing the Cost Function

To build intuition, let's visualize how different values of w affect the cost function J. Consider a simplified model where b = 0, so fw(x) = wx.

1. When w = 1:

- The line fw(x) = x perfectly fits the training data points (1,1), (2,2), and (3,3).

- The cost J(1) = 0 since the predicted values match the actual values exactly.

2. When w = 0.5:

- The line fw(x) = 0.5x doesn't fit the training data perfectly.

- The cost J(0.5) is higher because the predicted values are farther from the actual values.

3. When w = 0 :

- The line is horizontal at y = 0, which is a poor fit for the data.

- The cost J(0) is even higher.

By plotting these costs for various values of w, we obtain a curve. The goal is to find the w that minimizes this curve, indicating the best-fit line for the data.

Finding the Optimal Parameters

In practice, linear regression involves finding the parameters w and b that minimize the cost function J(w, b). This process ensures that the model makes accurate predictions for the training data and generalizes well to new, unseen data.

Understanding and visualizing the cost function helps in grasping the essence of machine learning optimization. By minimizing the cost function, we fine-tune our model to make it as accurate as possible.

Also read this blog by Li Yin: A Walk-through of Cost Functions-Medium blogs