How to handle null values and outliers

One of the many decisions you must make as a data analyst is handling null values and outliers.

Null values are data points that are missing or empty. They can occur for various reasons, such as human error, data collection issues, or technical problems.

Outliers are data points that fall outside of the normal range of values. Errors, unusual events, or fraudulent activity can cause them.

Both null values and outliers can have a significant impact on data analysis. If they are not handled properly, they can lead to inaccurate or misleading results.

Here are some tips on how to handle null values and outliers:

Null values

• Identify the null values. The first step is to identify all of the null values in your dataset. This can be done using a statistical software program or simply visually inspecting the data.
• Understand why the null values exist. Once you have identified the null values, it is important to try to understand why they exist. This will help you to decide how to handle them.
• Remove the null values. Removing them from the dataset is usually best if the null values are due to human error or data collection issues.
• Impute the null values. If the null values are due to technical problems or if they are important to the analysis, you may want to impute them. This means replacing the missing values with estimated values. There are a variety of imputation methods available, such as mean imputation, median imputation, and regression imputation.

Outliers

• Identify the outliers. The first step is to identify all of the outliers in your dataset. This can be done by using a statistical software program or by creating a boxplot or histogram of the data.
• Understand why the outliers exist. Once you have identified the outliers, understanding why they exist is important. This will help you to decide how to handle them.
• Remove the outliers. If the outliers are due to errors or fraudulent activity, removing them from the dataset is usually best.
• Transform the outliers. If the outliers are important to the analysis, you can transform them. This means converting the outlier values to values that are more consistent with the rest of the data. Various transformation methods are available, such as log and square root transformations.

Using domain knowledge to handle null values and outliers

Domain knowledge is essential for handling null values and outliers effectively. With domain knowledge, you can better understand what the values mean and how they may impact your data analysis.

For example, if you are analyzing a dataset of customer purchase data, you might know that customers are more likely not to provide their email address if they make a small purchase. This knowledge would help you to understand that the email address variable is MAR.

You could then decide to handle the missing email addresses by imputing them with the mean or median email address. However, if you know that customers who do not provide their email addresses are less likely to be repeat customers, you may remove them from the dataset altogether.

Domain knowledge can also help you to identify legitimate outliers. For example, if you are analyzing a dataset of sales data, you might know that a particular customer is a high roller and often places large orders. If you see a data point that shows that the customer placed a very large order, you would know that this is a legitimate data point, even though it is an outlier.

By using your domain knowledge, you can handle null values and outliers in a way that preserves the accuracy and integrity of your data analysis.

Types of missing values

There are three main types of missing values:

• MCAR (missing completely at random): This means that the probability of a missing data point is unrelated to the data point's value or any other variables in the dataset.
• MNAR (missing not at random): This means that the probability of missing data points depends on the data point's value or some other variable in the dataset. For example, if customers are more likely not to provide their email address if they have made a small purchase, then the email address variable would be MNAR.
• MAR (missing at random): This means that the probability of a data point being missing is unrelated to the value of the data point, but it may be related to other variables in the dataset. For example, if customers are more likely not to provide their email address if they are male, then the email address variable would be MAR.

Handling null values

The best way to handle null values depends on the type of missing values and the goals of the analysis. For MCAR and MAR data, imputation methods such as mean, median, and regression imputations can be used to fill in the missing values. For MNAR data, imputation methods may not be effective, and removing the rows or columns with missing values from the dataset may be necessary.

Handling outliers

Outliers can also be handled in a variety of ways. One common approach is to remove the outliers from the dataset. However, this can reduce the power of the analysis and may lead to biased results. Another approach is to transform the outliers using a method such as log transformation. This can make the outliers less influential in the analysis without removing them completely.

Using domain knowledge to handle null values and outliers

Domain knowledge is essential for handling null values and outliers effectively. With domain knowledge, you can better understand what the values mean and how they may impact your data analysis.

For example, if you are analyzing a dataset of customer purchase data, you might know that customers are more likely not to provide their email address if they make a small purchase. This knowledge would help you to understand that the email address variable is MAR. You could then decide to handle the missing email addresses by imputing them with the mean or median email address. However, if you know that customers who do not provide their email addresses are less likely to be repeat customers, you may remove them from the dataset altogether.

Domain knowledge can also help you to identify legitimate outliers. For example, if you are analyzing a dataset of sales data, you might know that a particular customer is a high roller and often places large orders. If you see a data point that shows that the customer placed a very large order, you would know that this is a legitimate data point, even though it is an outlier.

Using the missingno package in Python

The missingno package is a Python library that provides various tools for handling missing values. The missingno.matrix() function creates a heatmap that visualizes the missing values in a dataset. The missingno.heatmap() function creates a heatmap that shows the correlation between the missing values in different variables.

You can use the missingno package to help you identify and understand the missing values in your dataset. Once you have identified the missing values, you can use the information you have gained to decide how to handle them.

Treating missing values as an ML problem and subsampling

Another way to handle missing values is to treat it as an ML problem and subsample. This means that you would create a new dataset by randomly selecting a subset of the original dataset. The subset should be chosen to distribute the missing values similarly to the original dataset.

You can then use a machine learning algorithm to predict the missing values in the subset. Once the missing values have been predicted, you can merge the subset with the original dataset to create a complete dataset.

Subsampling can be a useful way to handle missing values, but it is important to note that it can also reduce the power of the analysis. Therefore, it is important to evaluate the results of the analysis carefully.

Additional tips for handling null values and outliers

• Use a variety of methods to identify null values and outliers. This will help you to get a more complete picture of the data.
• Document your decisions about how to handle null values and outliers. This will help you and others to understand and reproduce your analysis.
• Be transparent about the limitations of your data analysis. If you have removed null values or outliers, be sure to note this in your report.

By following these tips, you can handle null values and outliers in a way that produces accurate and reliable results.