Regression & Classification

Regression and classification are two predictive modeling approaches in statistics and machine learning. Here's a brief definition of each:

Regression

Regression analysis is a statistical method used for estimating the relationships among variables. It focuses on predicting a continuous outcome variable (the dependent variable) based on one or more predictor variables (independent variables).

Objective: Predict continuous value. The aim is to estimate the mapping function (f) from input variables (X) to a continuous output variable (Y).

Key Points: The output variable in the regression is continuous, which means it can take any value within a range. Common types of regression include linear regression, where the relationship between the variables is assumed to be linear, and non-linear regression, where the relationship can be more complex. Regression is often used for forecasting, determining the strength of predictors, and trend analysis.

Evaluation Metrics: Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), etc.

Algorithms: Linear Regression, Polynomial Regression (different orders), Ridge Regression, Lasso Regression, etc.

Example:

Simulating the Soil Organic Carbon and Total Nitrogen Relationship through Regression Analysis (This is Simulated Data)

Here I simulated a relationship between Soil Organic Carbon (SOC) and Total Nitrogen (TN) to demonstrate how regression analysis works in soil science. SOC and TN are closely linked in soil because organic carbon is a key component that binds nitrogen in soil. This simulation with hypothetical data reflects this strong correlation often observed in real-world soil studies. We found that the relationship between SOC and TN is very close, as indicated by an R-squared value >0.90.

# Load necessary libraries
if (!require("ggplot2")) install.packages("ggplot2")
library(ggplot2)
if (!require("ggpmisc")) install.packages("ggpmisc")
library(ggpmisc)

# Adjusting the simulated data
set.seed(123) # for reproducibility
n <- 100 # number of observations
soil_organic_carbon <- runif(n, min=1, max=5) # SOC in percentage

# Adjusting TN to be more closely related to SOC
total_nitrogen <- soil_organic_carbon * 0.18 + rnorm(n, mean=0, sd=0.05) # TN in percentage

# Recreate the dataframe
soil_data <- data.frame(soil_organic_carbon, total_nitrogen)

# Re-running the Linear Regression Model
model <- lm(total_nitrogen ~ soil_organic_carbon, data=soil_data)
summary(model)

#summary
Call:
lm(formula = total_nitrogen ~ soil_organic_carbon, data = soil_data)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.111899 -0.030661 -0.000987  0.029817  0.110861 

Coefficients:
                     Estimate Std. Error t value Pr(>|t|)    
(Intercept)         0.0006749  0.0136809   0.049    0.961    
soil_organic_carbon 0.1788771  0.0042728  41.864   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.04846 on 98 degrees of freedom
Multiple R-squared:  0.947,	Adjusted R-squared:  0.9465 
F-statistic:  1753 on 1 and 98 DF,  p-value: < 2.2e-16        

Explaining the summary of the model (R output):

1. Call:lm(formula = total_nitrogen ~ soil_organic_carbon, data = soil_data)This indicates the use of a linear model (lm) where total_nitrogen is predicted based on soil_organic_carbon using the dataset soil_data.
2. Residuals:The summary of residuals (differences between observed and predicted values) shows the spread of the residuals.Min, 1Q (first quartile), Median, 3Q (third quartile), and Max provide a five-number summary of the residuals. Ideally, the median should be close to zero, and the spread should be relatively small, indicating a good fit.
3. Coefficients:(Intercept) and soil_organic_carbon are the coefficients estimated by the model.Estimate: The coefficient value. For soil_organic_carbon, it's 0.1788771, indicating the change in total_nitrogen for a one-unit change in soil_organic_carbon.Std. Error: Standard error of the estimate, indicating the accuracy of the coefficient estimate.t value: The ratio of the coefficient to its standard error. A higher absolute value generally indicates a more significant coefficient. Pr(>|t|): P-value for testing the hypothesis. A small p-value (here < 2e-16) suggests that the variable is a significant predictor of the dependent variable.
4. Significance codes:Indicate the level of significance of the coefficients. Here, *** implies a high level of significance.
5. Residual standard error:It gives an estimate of the standard deviation of the residuals, 0.04846 in this case.
6. R-squared values:Multiple R-squared: 0.947, indicating that approximately 94.7% of the variance in total_nitrogen is explained by soil_organic_carbon.Adjusted R-squared: 0.9465, adjusted for the number of predictors, which is very close to the R-squared and indicates a strong model.
7. F-statistic:This test measures the overall significance of the model. A high F-value (1753 in this case) and a very small p-value (less than 2.2e-16) suggest that the model is statistically significant.

Visualization

# Re-plot with Regression Line, Equation, and R-Squared Value
ggplot(soil_data, aes(x=soil_organic_carbon, y=total_nitrogen)) +
  geom_point() +
  geom_smooth(method="lm", color="blue") +
  stat_poly_eq(
    formula = y ~ x, 
    aes(label = after_stat(paste(..eq.label.., ..rr.label.., sep = "~~~"))), 
    parse = TRUE
  ) +
  ggtitle("Soil Organic Carbon vs Total Nitrogen") +
  xlab("Soil Organic Carbon (%)") +
  ylab("Total Nitrogen (%)")        

Classification

Classification is the process of categorizing or classifying an item into a predefined set of categories or classes. It involves building a model that assigns new observations to one of several classes based on the features of the data.

Objective: To predict a categorical value. The aim is to estimate the mapping function (f) from input variables (X) to discrete output variables (Y).

Key Points: The output variable in classification is categorical, not numeric. It can be binary (e.g., yes/no, spam/not spam) or multi-class (e.g., types of fruits, categories of diseases). Common algorithms used for classification include logistic regression (despite its name, it's a classification method), decision trees, support vector machines, and neural networks. Classification is widely used in applications like email filtering (spam or not spam), medical diagnosis, image recognition, and more.

Evaluation Metrics: Accuracy, Precision, Recall, F1-Score, Confusion Matrix, ROC Curve, etc.

Algorithms: Logistic Regression (used for classification), Decision Trees, Support Vector Machines (SVM), Neural Networks, etc.

Example:

Using hypothetical simulated data, our classification example focused on distinguishing different soil types based on their physical and chemical properties. Specifically, we simulated data for soil characteristics like organic matter percentage, clay content, and drainage quality and categorized the soil into different types such as Sand, Loam, and Clay.

Type of Classification Used

We used a Support Vector Machine (SVM) model for our simulation. SVM is a popular machine learning algorithm used for classification tasks. It works well for both linear and non-linear classification. In our case, SVM helped to classify soil into categories based on the given features (organic matter, clay content, and drainage quality). SVM is particularly effective in cases where the relationship between the class and features is not straightforward or when the data isn't linearly separable.

# Load necessary library
if (!require("e1071")) install.packages("e1071")
library(e1071)

# Simulating data
set.seed(123)
n <- 100 # number of observations
organic_matter <- rnorm(n, mean=30, sd=10) # organic matter in percentage
clay_content <- rnorm(n, mean=35, sd=15) # clay content in percentage
drainage_quality <- runif(n, min=1, max=5) # drainage quality on a scale of 1 to 5

# Simulating soil types (e.g., Sand, Loam, Clay)
soil_types <- as.factor(sample(c("Sand", "Loam", "Clay"), n, replace=TRUE))

# Combining into a dataframe
soil_class_data <- data.frame(organic_matter, clay_content, drainage_quality, soil_types)

# SVM Classification Model
model <- svm(soil_types ~ organic_matter + clay_content + drainage_quality, data=soil_class_data)
summary(model)        

#Summary output
Call:
svm(formula = soil_types ~ organic_matter + clay_content + drainage_quality, data = soil_class_data)


Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  radial 
       cost:  1 

Number of Support Vectors:  96

 ( 37 35 24 )


Number of Classes:  3 

Levels: 
 Clay Loam Sand        

Here,

1. Call:svm(formula = soil_types ~ organic_matter + clay_content + drainage_quality, data = soil_class_data)This shows the SVM model was used to classify soil_types based on three predictors: organic_matter, clay_content, and drainage_quality, using the dataset soil_class_data.
2. Parameters:SVM-Type: C-classification. This indicates that the SVM is used for a classification problem.SVM-Kernel: radial. The radial basis function (RBF) kernel is used. This is a common choice for classification and can handle non-linear relationships between features.cost: 1. This is the regularization parameter. It controls the trade-off between achieving a low error on the training data and minimizing the model complexity for better generalization.
3. Number of Support Vectors: 96 (37 35 24)A total of 96 support vectors were used in the model. The numbers in parentheses (37, 35, 24) represent the number of support vectors for each class. If you have three classes (like Clay, Loam, and Sand), these numbers correspond to the support vectors for each of these classes respectively.
4. Number of Classes: 3This indicates that the model is differentiating between the three classes.
5. Levels:Clay, Loam, and Sand are the class levels (categories) into which the model classifies the data.

Visualization of Classification

We visualized this classification using a scatter plot in R, where points with different colors represented the different soil types. Additionally, we enhanced the plot by drawing ellipses around each group of soil types using the stat_ellipse function. This helped visually distinguish the clusters of different soil types based on the given features. This classification example showcases the practical application of machine learning in soil science. Understanding and categorizing soil types based on various properties is vital for agricultural planning, environmental assessment, and soil management.

ggplot(soil_class_data, aes(x=organic_matter, y=clay_content, color=soil_types)) +
  geom_point() +
  stat_ellipse(type="t", linetype=2) +
  ggtitle("Soil Type Classification") +
  xlab("Organic Matter (%)") +
  ylab("Clay Content (%)")        

Key Differences

1. Output Type: Regression predicts a continuous value, while classification predicts a categorical value.
2. Problem Nature: Regression is used to predict a value, whereas classification is used to separate data into classes.
3. Evaluation: Different metrics are used to evaluate the performance of regression and classification models.
4. Complexity: Classification problems can sometimes be more complex due to separating data into distinct categories, especially in multi-class scenarios.