Handling Missing Data

Data is the foundation of any analytical project. However, real-world data is often messy and incomplete. Missing data can introduce bias, reduce the power and efficiency of statistical methods/models, and complicate analyses. Handling missing data is therefore a critical step in ensuring the accuracy and reliability of your results.

Why Does Data Go Missing?

Before we go deep into the methods of handling missing data, it's important to understand why data might be missing. Some common reasons include:

1. Errors during data collection: Mistakes can occur during data entry or data transfer.
2. Non-responses: Survey respondents might skip certain questions.
3. Data decay: Over time, some data points might get lost or become inaccessible.
4. Systematic issues: Sometimes, missing data isn't random. For instance, a faulty sensor might not record any data.

Understanding the reason behind the missing data can guide you in selecting the best method to handle it.

Types of Missing Data

A. Missing Completely at Random (MCAR): The missing data has no relationship with any other data point. The causes of the missing data are random.

• Scenario: A soil researcher collects samples from multiple sites to analyze soil pH. While transporting the samples to the lab, some of the containers accidentally opened, and the samples were lost.
• Example: The loss of samples was purely accidental and had nothing to do with the pH or any other characteristic of the soil. This means the missingness of the pH data from these samples is completely random.

B. Missing at Random (MAR): The reason for the missing data is related to some other observed data but not the missing data itself.

• Scenario: A research team is studying various soil properties, including moisture content, pH, and nutrient levels, across different altitudes. They notice that at higher altitudes, it's more challenging to measure moisture content because the equipment malfunctions in colder temperatures.
• Example: The missingness of moisture content data is related to the altitude (which is observed data) but not directly to the moisture content itself. If we account for altitude in our analysis, we can potentially address the bias introduced by the missing moisture content data.

C. Missing Not at Random (MNAR): There's a specific reason for the missing data, and it's related to the missing data itself.

• Scenario: A study is being conducted on soil salinity in areas near the coast. Some farmers, knowing that high salinity levels in their fields could reduce land value, decline to provide soil samples from salt-affected patches.
• Example: Here, the missingness of the salinity data is directly related to the salinity level of the soil (the higher the salinity, the more likely the data is missing). The missingness mechanism is inherently related to the unobserved missing value itself, making it MNAR.

Techniques for Handling Missing Data

1. Deletion:Listwise: Remove any row with a missing value.Pairwise: Remove specific pairs of data where one or both are missing.Pros: Simple and quick.Cons: This can result in significant data loss.
2. Imputation:Mean/Median/Mode Imputation: Replace missing values with the mean, median, or mode of the column.Linear Regression Imputation: Predict missing values using linear regression.K-Nearest Neighbors (KNN) Imputation: Replace missing values with the mean or median of their k-nearest neighbors.Pros: Retains the original data size.Cons: Can introduce bias or reduce variance.
3. Interpolation: Useful for time series data. Predict missing values based on the values of surrounding data points. [I will discuss more about the pros/cons in the next post]
4. Multiple Imputation: Impute the missing data multiple times to create several complete datasets and then average the results. [I will discuss more about the pros/cons in the next post]
5. Using Algorithms that Support Missing Values: Some algorithms like XGBoost or certain tree-based algorithms can handle missing values without any preprocessing. [I will discuss more about the pros/cons in the next post]

Examples:

1. Missing Completely at Random (MCAR):

Scenario & Dataset:

• 10 soil samples were collected to measure pH.
• Due to an accident, the pH values of 2 samples were lost.

# Hypothetical dataset
set.seed(123)
soil_pH <- data.frame(Sample_ID = 1:10, pH = c(runif(8, 5, 9), NA, NA))
print(soil_pH)        

Solution:

• Given the data is MCAR, we can use mean imputation.

soil_pH$pH[is.na(soil_pH$pH)] <- mean(soil_pH$pH, na.rm = TRUE) print(soil_pH)        

2. Missing at Random (MAR):

Scenario & Dataset:

• Soil properties are measured across different altitudes.
• At higher altitudes, moisture content measurements are missing.

# Hypothetical dataset
altitude <- c(seq(100, 800, 100))
moisture_content <- c(20, 21, 22, 19, NA, NA, NA, NA)  
# Missing data at higher altitudes
soil_data_mar <- data.frame(Altitude = altitude, Moisture_Content = moisture_content)
print(soil_data_mar)        

Solution:

• Given the data is MAR and there's a trend (lower moisture with higher altitude), we can use linear regression imputation.

model <- lm(Moisture_Content ~ Altitude, data = soil_data_mar, na.action = na.exclude)
soil_data_mar$Moisture_Content[is.na(soil_data_mar$Moisture_Content)] <- predict(model, newdata = soil_data_mar[is.na(soil_data_mar$Moisture_Content), ])
print(soil_data_mar)        

3. Missing Not at Random (MNAR):

Scenario & Dataset:

• Soil salinity is measured in areas near the coast.
• Farmers with high salinity levels decline to provide samples.

# Hypothetical dataset distance_from_coast <- c(seq(1, 10, 1)) salinity <- c(3, 5, 4, 6, 8, NA, NA, NA, NA, NA) # Missing data for fields closer to the coast soil_data_mnar <- data.frame(Distance = distance_from_coast, Salinity = salinity) print(soil_data_mnar)        

Solution:

• MNAR is tricky. One could use advanced methods like multiple imputations. But for simplicity, let's use KNN imputation.

library(DMwR) 
soil_data_mnar_imputed <- DMwR::knnImputation(soil_data_mnar, k = 3) print(soil_data_mnar_imputed)        

Note: Solutions to missing data problems always come with their assumptions and limitations. Especially for MNAR, there's no perfect solution. In real-world scenarios, domain expertise and understanding of the context play a significant role in choosing the most appropriate method.

Few more examples:

A hypothetical dataset:

set.seed(123)        

Soil_Depth <- runif(100, 0, 100)  # Depth in cm        

Carbon_Content <- 0.5 * Soil_Depth + rnorm(100, mean=0, sd=5)        

Carbon_Content[sample(1:100, 15)] <- NA  # Introducing missing values        

soil_data <- data.frame(Soil_Depth, Carbon_Content)        

visualizing the original dataset:

ggplot(soil_data, aes(Soil_Depth, Carbon_Content)) + 
  geom_point(na.rm = TRUE) + 
  geom_point(data = soil_data[is.na(soil_data$Carbon_Content), ], aes(Soil_Depth, Carbon_Content), color = "red") + 
  ggtitle("Soil Depth vs. Organic Carbon Content")        

Listwise deletion:

soil_data_listwise <- soil_data[complete.cases(soil_data), ]        

ggplot(soil_data_listwise, aes(Soil_Depth, Carbon_Content)) + geom_point() + ggtitle("Listwise Deletion")        

Mean imputation:

soil_data_mean <- soil_data        

soil_data_mean$Carbon_Content[is.na(soil_data_mean$Carbon_Content)] <- mean(soil_data_mean$Carbon_Content, na.rm = TRUE)        

ggplot(soil_data_mean, aes(Soil_Depth, Carbon_Content)) + geom_point() + ggtitle("Mean Imputation")        

Linear Regression Imputation:

model <- lm(Carbon_Content ~ Soil_Depth, data = soil_data, na.action = na.exclude)        

soil_data_regression <- soil_data        

soil_data_regression$Carbon_Content[is.na(soil_data_regression$Carbon_Content)] <- predict(model, newdata = soil_data_regression[is.na(soil_data_regression$Carbon_Content), ])        

ggplot(soil_data_regression, aes(Soil_Depth, Carbon_Content)) + geom_point() + ggtitle("Linear Regression Imputation")        

KNN:

soil_data_knn <- DMwR::knnImputation(soil_data, k = 5)        

ggplot(soil_data_knn, aes(Soil_Depth, Carbon_Content)) + geom_point() + ggtitle("KNN Imputation")        

___________________________________________________________________

I will update this post with a time series analysis and another test next week. Like temporal decomposition, additive models to predict the missing values, linear interpolations, and RNNs.