Handling Missing Values: Effective Strategies and Techniques

In any dataset, missing values refer to the absence of data for a particular variable in some observations. This means that for certain records or data points, there's no information stored for one or more variables. Missing data can arise from a variety of sources, and it can significantly impact the quality and reliability of data analysis. When data is missing, the analysis might become less accurate, leading to incorrect or misleading conclusions.

Causes of Missing Data

1. Human Error: Mistakes made by individuals during data collection or entry, such as accidentally skipping a question in a survey or entering incorrect values.

2. Data Entry Errors: Errors that happen during the process of inputting data into a system, such as misplacing decimal points or omitting information.

3. Equipment Failures: Technical issues, such as malfunctioning sensors or broken instruments, that prevent data from being recorded.

4. Non-Response: In surveys or studies, participants may choose not to answer certain questions, leading to missing data for those variables.

Types of Missing Data

When dealing with missing data, it's important to understand the different types and how they can affect analysis. There are three main categories:

1. Missing Completely at Random (MCAR)

When data is Missing Completely at Random (MCAR), the missingness occurs purely by chance and is unrelated to any observed or unobserved variables in the dataset. In this case, there's no systematic pattern behind the missing data, and it is independent of the data itself.

Example: Imagine conducting a survey where some participants skip questions randomly without any specific reason. For instance, one person might forget to answer a question simply because they were distracted at the time, while another might accidentally skip a question. Since these omissions are random and not related to any particular characteristics of the respondents or the questions, the data is considered MCAR.

Impact: MCAR is the least problematic type of missing data because the randomness means it doesn’t introduce bias into the analysis. Statistical techniques can be used to handle MCAR data effectively, such as listwise deletion (removing cases with missing data) or imputing (filling in) the missing values using simple methods like the mean or median of the observed data.

2. Missing at Random (MAR)

When data is Missing at Random (MAR), the probability of a value being missing is related to the observed data but not to the missing data itself. In other words, the missingness is systematic, but it is associated only with the observed variables and not with the values that are missing.

Example: Consider a clinical study where younger participants are less likely to report their income. Here, the likelihood of missing income data is related to the age of the participants (an observed variable). However, the missingness is not related to the actual income itself, meaning that among younger participants, whether or not they report their income is unrelated to how much they earn. This situation would be classified as MAR.

Impact: MAR can introduce bias into the analysis if not handled properly, as the missingness is related to other variables in the dataset. However, there are statistical methods designed to adjust for this bias. For example, multiple imputation or model-based methods can be used to predict the missing values based on the observed data, helping to reduce the impact of MAR on the analysis.

3. Missing Not at Random (MNAR)

When data is Missing Not at Random (MNAR), the probability of missingness is related to the unobserved data itself. This means that the reason data is missing is directly linked to the value that is missing. This type of missing data is the most challenging to deal with because the missingness is not random, and it is systematically related to the missing values.

Example: In a study on mental health, individuals with more severe symptoms might be less likely to report their condition, possibly due to stigma or fear of judgment. As a result, the missing data is directly related to the severity of the symptoms—the more severe the symptoms, the more likely the data is missing. This scenario would be considered MNAR.

Impact: MNAR is the most difficult type of missing data to handle because it introduces significant bias into the analysis. Since the missingness is related to the unobserved values, standard methods like imputation may not be sufficient to correct for the bias. In such cases, more advanced techniques, such as modeling the missing data mechanism, may be necessary to properly adjust the analysis and draw valid conclusions.

Understanding the types of missing data and their implications is crucial for conducting reliable data analysis. While MCAR data can be dealt with using relatively simple techniques, MAR and MNAR require more sophisticated methods to address the potential biases they introduce. Properly handling missing data is essential to ensure the accuracy and validity of the conclusions drawn from any dataset.

These are the things to consider before treating missing values

1. Understand the Missing Data Mechanism: Before choosing a method to handle missing data, it’s crucial to understand the nature of the missingness (MCAR, MAR, MNAR) and its potential impact on the analysis.

2. Use Multiple Methods: When feasible, apply multiple methods and compare the results to ensure robustness.

3. Document and Report: Clearly document the presence of missing data, the methods used to handle it, and any assumptions made. Reporting the extent of missing data and its handling is essential for transparency and reproducibility.

This is the dataset we'll be working with for this demonstration, I have generated this dataset using Python, the python code along with the dataset is available to download. link is given at the bottom of this article.

Here are some imputation techniques we can use for handling missing values.

1. Mean Imputation

What it is: This method replaces missing values in a numerical column (like Age or Salary) with the average value of that column.

Why use it: It's simple and works well when missing values are spread evenly and are not too many.

Example: If the average age in the dataset is 40, all missing ages will be replaced with 40.

Code:

data['Age'].fillna(data['Age'].mean(), inplace=True)

data['Salary'].fillna(data['Salary'].mean(), inplace=True)

Output:

As you can see, the missing values are replaced by mean of the Age & Salary columns. Mean imputation method is limited to perform imputation only on numerical columns. it does not support imputation on string columns. so you'll see this similar outcomes with the other method below as well.

2. Median Imputation

What it is: Similar to mean imputation, but it replaces missing values with the median (middle) value of the column.

Why use it: It's better than mean imputation when your data has outliers (extremely high or low values) because the median is less affected by them.

Example: If the median salary is $50,000, all missing salaries will be replaced with $50,000.

Code:

data['Age'].fillna(data['Age'].median(), inplace=True)

data['Salary'].fillna(data['Salary'].median(), inplace=True)

Output:

As you can see, The Null values are imputed with the median value of the Age & Salary columns. similar to mean imputation, median imputation also does not support imputation of string columns.

3. Mode Imputation

What it is: This method is used for categorical data (like Gender or Purchase History). It replaces missing values with the most frequent value (the mode).

Why use it: It’s useful when you want to fill in missing categories with the most common category.

Example: If most people in the dataset are female, all missing genders would be filled with "Female."

Code:

data['Age'].fillna(data['Age'].mode()[0], inplace=True)

data['Salary'].fillna(data['Salary'].mode()[0], inplace=True)

data['Gender'].fillna(data['Gender'].mode()[0], inplace=True)

data['Purchase History'].fillna(data['Purchase History'].mode()[0], inplace=True)

Output:

Mode imputation fills the empty records with the most repeating value in the column, it can be helpful in variety of scenarios when you are faced with missing string column data. Unlike mean and median imputations, mode imputation supports imputation of both the numerical and string columns as perfectly demonstrated by the above image.

4. Constant Imputation

What it is: You replace missing values with a specific number or text that you choose.

Why use it: This is handy when you want to indicate missing data with a placeholder value or if there's a logical default value.

Example: If a salary is missing, you could set it to $0 or any default amount you choose. If purchase history is missing, you could fill it with "Unknown."

Code:

data['Age'].fillna(0, inplace=True)

data['Salary'].fillna(50000, inplace=True)

data['Gender'].fillna('Unknown', inplace=True)

data['Purchase History'].fillna('Unknown', inplace=True)

Output:

Similar to mode imputation, Constant imputation also supports imputation of both the numerical and string columns, but we have to specify which value that needs to be imputed. The above image clearly demonstrates the imputed values of our required conditions which we have set in the code.

5. Forward Fill

What it is: This method fills in missing values by using the last available value in the column.

Why use it: It's useful in time-series data or sequences where the previous value is likely to continue.

Example: If a person’s age is missing in the next row but was 30 in the previous row, it will be filled with 30.

Code:

data.fillna(method='ffill', inplace=True)

Output:

This forward fill method imputes the missing record with the previous row, i.e the second row of the Age column was previously empty and this method imputed the empty row with the value of the previous row.

6. Backward Fill

What it is: Opposite to forward fill, this method uses the next available value to fill in missing ones.

Why use it: Similar to forward fill, but instead of using previous values, it uses upcoming ones.

Example: If salary is missing but is known to be $60,000 in the next row, the missing salary will be filled with $60,000.

Code:

data.fillna(method='bfill', inplace=True)

Output:

Unlike forward fill, backward fill method imputes values based on the next record. like previously, the second row of the age column was empty and it got filled with the value of the next record i.e the 3rd row.

7. K-Nearest Neighbors (KNN) Imputation

What it is: This advanced method uses the values of nearby data points (neighbors) to estimate and fill in the missing values.

Why use it: It considers relationships between variables and can be more accurate than simple methods like mean or median imputation.

Example: If a person’s age is missing, KNN will look at people with similar attributes (like similar salary and gender) and use their ages to estimate the missing one.

Code:

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)

data[['Age', 'Salary']] = imputer.fit_transform(data[['Age', 'Salary']])

Output:

As demonstrated by the above image, KNN imputation had imputed the missing values by the nearest data point. similar to mean and median imputation, this only supports imputation on numerical columns.

8. Multivariate Imputation by Chained Equations (MICE)

What it is: This method repeatedly fills in missing values by predicting them based on other variables in the dataset.

Why use it: It handles complex data relationships and provides a robust way to deal with multiple missing values across columns.

Example: If both age and salary are missing, MICE will predict each missing value iteratively using the other variables.

Code:

from sklearn.experimental import enable_iterative_imputer

from sklearn.impute import IterativeImputer

imputer = IterativeImputer(max_iter=10, random_state=0)

data[['Age', 'Salary']] = imputer.fit_transform(data[['Age', 'Salary']])

Output:

As demonstrated from the above image, MICE imputation filled the missing values by predicting values based on other variables that are present in the dataset.

Conclusion

In conclusion, choosing the best way to handle missing data is like picking the right tool for a job at home. Just as you wouldn't use a hammer to tighten a screw, you need to select the right method based on your specific data and what you're trying to achieve. Each approach, like forward fill or others, has its pros and cons, so understanding your data and your goals is key to making the best choice.

Note - Below are the tools and libraries were used. I have also added a link for you to download the files.

Framework - Spyder IDE, Language- Python, Libraries- Pandas, Scikit-Learn

Link - https://mega.nz/folder/7hpmBBzZ#pm22SPl_bQuTIw9hW_itLA