Evaluating Linear Regression Models

Linear regression is a powerful and commonly used technique in machine learning and statistics. It helps us understand the relationship between a dependent variable (the one we want to predict) and one or more independent variables (the ones we use for prediction). But how do we know if our linear regression model is doing well? In this article, we'll explore how to measure the performance of a linear regression model with Python using a practical example.

Note: To learn more about Linear Regression, you can read the following article:

Understanding the Data

For our example, we will work with a dataset that contains information about houses, such as square footage, the number of bedrooms and bathrooms, the number of offers made, and other factors that can influence the price of a house. You can download the dataset from this link. Let's load the data and explore it a bit:

import pandas as pd

# Load the dataset
csv_url = "https://raw.githubusercontent.com/ywchiu/riii/master/data/house-prices.csv"
data = pd.read_csv(csv_url)

# Let's take a look at the first few rows of the data
data.head()
        

Preparing the Data

Before we can build a linear regression model, we need to prepare the data. This involves converting categorical variables into a numerical format, and defining our independent (features) and dependent (target) variables.

One-Hot Encoding for Categorical Variables

Categorical variables like 'Brick' and 'Neighborhood' need to be converted into a numerical format. Categorical variables like 'Brick' and 'Neighborhood' are converted to a numerical format, often using one-hot encoding, for several important reasons:

1. Numerical Representation: Machine learning algorithms, including linear regression, are based on mathematical equations. They require numerical inputs. Categorical variables, such as 'Brick' with categories like 'Yes' and 'No' or 'Neighborhood' with categories like 'North' and 'West', cannot be used directly in these equations. Converting them to a numerical format makes it possible to include them as features in the model.
2. Avoiding Ambiguity: Converting categorical variables to numerical format helps avoid any ambiguity in the model. For instance, if we left 'Brick' as 'Yes' or 'No', the model might interpret it as a numeric scale (e.g., 1 for 'Yes' and 0 for 'No'), potentially leading to incorrect conclusions.
3. Improved Model Performance: Converting categorical variables to numerical format can often improve the performance of the model. It provides a more accurate representation of the relationships between the variables and the target variable.
4. Handling Multiple Categories: If a categorical variable has multiple categories (e.g., 'Neighborhood' with several neighborhoods), one-hot encoding creates binary columns for each category. This is important because it allows the model to differentiate between different categories without imposing any ordinal relationship between them.
5. Interpretable Coefficients: When you use numerical encoding, the coefficients associated with each category (e.g., 'Brick_Yes' or 'Neighborhood_North') in the linear regression model represent the change in the target variable associated with that category while keeping other factors constant. This makes the model results more interpretable.
6. Compatibility with Algorithms: Many machine learning algorithms, including scikit-learn's Linear Regression, expect numerical input. By converting categorical variables to a numerical format, you ensure compatibility with these algorithms.

Keep in mind that one-hot encoding, which creates binary (0 or 1) columns for each category, is just one method to convert categorical variables. Other techniques, such as label encoding or ordinal encoding, may be used depending on the nature of the categorical data and the specific problem you are trying to solve.

We can achieve this using one-hot encoding. Here's how we do it:

# Convert categorical variables to numerical using one-hot encoding
data = pd.get_dummies(data, columns=['Brick', 'Neighborhood'], drop_first=True)

data.head()        

Now, our dataset contains numerical representations of these categorical variables.

Defining Features and Target

We need to define our independent variables (features) and our dependent variable (target). In this case, we want to predict the house 'Price' based on other features. Here's how we do it:

# Define the independent variables (features)
X = data[['SqFt', 'Bedrooms', 'Bathrooms', 'Offers', 'Brick_Yes', 'Neighborhood_North', 'Neighborhood_West']]

# Define the dependent variable
Y = data['Price']
        

Now we're ready to build and evaluate our linear regression model.

Building and Evaluating the Model

We'll use Python's scikit-learn library to build and evaluate our linear regression model. Here's the code:

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_squared_error

# Create a Linear Regression model
model = LinearRegression()

# Fit the model to our data
model.fit(X, Y)

# Predict house prices using the model
predicted_prices = model.predict(X)

# Calculate R-squared (R^2)
r_squared = r2_score(Y, predicted_prices)

# Calculate Mean Squared Error (MSE)
mse = mean_squared_error(Y, predicted_prices)

# Print model coefficients
coefficients = model.coef_
intercept = model.intercept_
print("Model Coefficients:")
for feature, coef in zip(X.columns, coefficients):
    print(f"{feature}: {coef:.2f}")
print(f"\nIntercept: {intercept:.2f}")
print(f"R-squared (R^2): {r_squared:.2f}")
print(f"Mean Squared Error (MSE): {mse:.2f}")
        

Let's break down what we've done:

• We created a Linear Regression model and fit it to our data using model.fit(X, Y).
• We made predictions using the model and calculated two important metrics: R-squared (R^2) and Mean Squared Error (MSE).
• R-squared tells us how well our model explains the variation in house prices. An R^2 value of 1 means a perfect fit, while lower values indicate less accurate predictions.
• MSE measures the average squared difference between our predicted and actual prices. Smaller values are better, indicating more accurate predictions.

Evaluating the Model:

Measuring the performance or accuracy of a multiple regression model with multiple features involves using various evaluation metrics to assess how well the model fits the data and makes predictions. Here are some common methods to evaluate the performance of a multiple regression model:

R-squared (R^2):

• R-squared is a widely used metric that quantifies the proportion of the variance in the dependent variable explained by the independent variables in the model. It ranges from 0 to 1, with higher values indicating a better fit.
• A high R-squared value suggests that the model explains a large portion of the variance in the target variable, while a low value indicates that the model doesn't explain much of the variance.
• However, R-squared alone may not provide a complete picture of model performance, especially in the presence of overfitting. Therefore, we may use the Adjusted R-squared (Adjusted R^2).
• Here is how to calculate R^2 using Python:

# Calculate R-squared (R^2)
r_squared = r2_score(Y, predicted_prices)        

Adjusted R-squared (Adjusted R^2):

• Adjusted R-squared is a modification of R-squared that adjusts for the number of independent variables in the model. It accounts for model complexity and is useful when comparing models with different numbers of features.A higher adjusted R-squared value is preferable, as it indicates a better balance between model complexity and explanatory power. To calculate the adjusted R-squared (adjusted R^2) using scikit-learn, you can use the following code. Scikit-learn does not provide a direct method to calculate adjusted R-squared, so you'll need to calculate it manually based on the R-squared value and the number of features.

# Calculate R-squared (R^2)
r_squared = model.score(X, Y)

# Calculate the total number of data points and the number of independent variables
n = len(Y)  # Total number of data points
k = X.shape[1]  # Number of independent variables

# Calculate Adjusted R-squared
adjusted_r_squared = 1 - (1 - r_squared) * (n - 1) / (n - k - 1)

print("R-squared:", r_squared)
print("Adjusted R-squared:", adjusted_r_squared)        

R-squared: 0.8686210289688724
Adjusted R-squared: 0.8609572556587233        

• You calculate R-squared using the model.score(X, Y) method, which returns the coefficient of determination (R-squared).
• You then calculate the adjusted R-squared using the formula:Adjusted R^2 = 1 - (1 - R^2) * (n - 1) / (n - k - 1)Where:n is the total number of data points.k is the number of independent variables (features).
• Finally, you print both R-squared and adjusted R-squared.

This code will provide you with the adjusted R-squared value using scikit-learn's LinearRegression model and the manually calculated adjusted R-squared based on the model's R-squared value and the number of features.

Why do we use Adjusted R^2 Over R^2?

R-squared (R^2) and adjusted R-squared (adjusted R^2) are both metrics used in regression analysis to evaluate the goodness of fit of a regression model. However, they serve slightly different purposes and have some key differences:

1. R-squared (R^2):R-squared is a measure of how well the independent variables (features) in a regression model explain the variation in the dependent variable.It is a value between 0 and 1, where 0 indicates that the model does not explain any of the variation, and 1 indicates that the model explains all of the variation.R-squared measures the proportion of the total variance in the dependent variable that is explained by the model. The formula is typically:R^2 = 1 - (RSS / TSS)Where:RSS (Residual Sum of Squares) is the sum of the squared residuals (errors) between the observed and predicted values.TSS (Total Sum of Squares) is the sum of the squared differences between the observed values and the mean of the dependent variable.R-squared tends to increase as more independent variables are added to the model, even if those variables do not truly improve the model's performance. This is because R-squared only increases as you add predictors, and it doesn't account for the possibility of overfitting.
2. Adjusted R-squared (Adjusted R^2):Adjusted R-squared is a modification of R-squared that takes into account the number of independent variables in the model.It is adjusted for the degrees of freedom in the model, which means it provides a more accurate assessment of the model's goodness of fit, particularly when you have multiple independent variables.The formula for adjusted R-squared is:Adjusted R^2 = 1 - [(1 - R^2) * (n - 1) / (n - k - 1)]Where:n is the number of data points (samples).k is the number of independent variables in the model.Adjusted R-squared will decrease if you add independent variables that do not improve the model, making it a more stringent measure of model fit compared to R-squared.

In summary, R-squared and adjusted R-squared both provide information about the goodness of fit of a regression model. R-squared is straightforward and can be influenced by the number of predictors, while adjusted R-squared adjusts for model complexity by considering the number of independent variables in the model. Adjusted R-squared is often preferred when comparing models with different numbers of predictors, as it penalizes the addition of irrelevant variables that do not improve model performance.

Insights on the previous numbers:

Let's explore what does this mean with the results we had for the house prices model:

R-squared: 0.8686210289688724
Adjusted R-squared: 0.8609572556587233        

The R-squared (R^2) and adjusted R-squared values provide important insights into the performance and validation of the linear regression model. Let's interpret these values:

1. R-squared (R^2): R-squared is a measure of how well the independent variables in the model explain the variation in the dependent variable (house prices, in this case). An R^2 value of 0.8686 means that approximately 86.86% of the variance in house prices can be explained by the independent variables included in the model. Here are some insights:A high R-squared value (close to 1) suggests that the model does a good job of explaining and predicting the target variable. In this case, the model explains a significant portion of the variability in house prices. It's a positive sign that the model has good explanatory power.
2. Adjusted R-squared (Adjusted R^2): Adjusted R-squared takes into account the number of independent variables in the model, providing a more conservative measure of model fit. An adjusted R^2 value of 0.8610 indicates that, while the model explains a substantial portion of the variance, it accounts for the model's complexity due to the multiple features:Adjusted R-squared is slightly lower than R-squared, as it penalizes the inclusion of unnecessary variables or overfitting. This suggests that, while the model is strong, it may benefit from some simplification or further refinement. It considers the trade-off between model complexity and explanatory power.

Interpreting these values in the context of model validation:

• The high R-squared indicates that the chosen independent variables provide valuable information for predicting house prices. It's a good sign that the model fits the data well.
• The adjusted R-squared provides a more conservative evaluation. In this case, it suggests that, while the model is strong, it may not need all the features included. There could be some room for optimization or feature selection to potentially improve the model's performance.
• Model validation is an ongoing process. You might consider experimenting with different feature combinations, testing the model's performance on a holdout dataset (cross-validation), and considering domain knowledge to fine-tune the model further.

Overall, these metrics show that the model performs well in explaining house prices but also indicates that there is potential for model improvement or simplification.

Other Methods of Evaluation:

Here are some other ways to evaluate the model

• Mean Squared Error (MSE):MSE measures the average of the squared differences between the observed and predicted values. It quantifies the model's accuracy in predicting the dependent variable.Lower MSE values indicate a better fit, as they represent smaller prediction errors.
• Root Mean Squared Error (RMSE):RMSE is the square root of the MSE and is expressed in the same units as the dependent variable. It provides a more interpretable measure of prediction error.Like MSE, lower RMSE values are better.
• Mean Absolute Error (MAE):MAE is the average of the absolute differences between the observed and predicted values. It measures the average magnitude of errors.Lower MAE values indicate better predictive accuracy.
• F-statistic and p-value:The F-statistic tests the overall significance of the regression model. It assesses whether at least one independent variable has a non-zero coefficient.The p-value associated with the F-statistic helps determine whether the model as a whole is statistically significant. A low p-value (typically below 0.05) suggests the model is significant.
• t-statistics and p-values for coefficients:You can evaluate the significance of each independent variable using t-statistics and their associated p-values. A low p-value for a coefficient indicates that the variable contributes significantly to the model.
• Residual Analysis:Plotting residuals (the differences between observed and predicted values) can help identify patterns or deviations in the model's predictions. Common residual plots include the scatterplot of residuals versus predicted values and a histogram of residuals to check for normality.
• Cross-Validation:Perform k-fold cross-validation to assess how well the model generalizes to unseen data. This helps evaluate the model's predictive performance and guard against overfitting.

When assessing the performance of a multiple regression model, it's often advisable to consider multiple metrics and techniques to gain a comprehensive understanding of how well the model fits the data and makes accurate predictions. The choice of evaluation metrics depends on the specific goals of the analysis and the characteristics of the dataset.

Visualizing the Model

To understand how our model performs visually, we can create scatter plots to compare actual prices to predicted prices. You can use libraries like Matplotlib to create these plots.

pythonCopy code        

import matplotlib.pyplot as plt # Create a scatter plot plt.scatter(Y, predicted_prices) plt.xlabel("Actual Prices") plt.ylabel("Predicted Prices") plt.title("Actual Prices vs. Predicted Prices") plt.show()

This scatter plot helps us see how well our model's predictions align with the actual prices. Ideally, the points should form a straight line.

Conclusion

Measuring the performance of a linear regression model is crucial in understanding how well it predicts outcomes. R-squared and Mean Squared Error are two essential metrics for this purpose. By following the steps outlined in this article and using Python, you can build, evaluate, and visualize a linear regression model, making data analysis and predictions more accessible to beginners and non-technical individuals. Linear regression provides a solid foundation for understanding more advanced machine learning techniques, so mastering it is an excellent starting point for anyone interested in the field.