What are the best practices for handling missing data in your data sets?

I don’t want to sound judgmental but it’s a shocking negligence. How did your data get missing? Why didn’t you put a tracker on it (we use airtags)? Did you try to put posters outside with a photo of it?

To identify missing data, you can use tools like summary statistics, visualizations, or Python libraries like pandas or missingno. Summary statistics calculate metrics like count, mean, and standard deviation to identify variables with fewer values than expected. Visualizations like bar charts or heatmaps can highlight gaps or patterns in the data that may indicate missing values. Pandas provides functions like isnull() or isna() to detect missing values, while missingno offers visualizations for missing data patterns. Examining metadata and documentation of the data source can explain reasons for missing data, such as data entry errors or non-response in surveys.

When analyzing or processing data, it's crucial to understand how missing values are coded in order to manage them correctly. Here are ways to identify missing values: ? NaN: Used in Python for "Not a Number". ? NULL: Used in databases for unassigned values. ? NA: Code for "Not Available" in R. ? -999, -9999: Out-of-range values indicating missing data in some contexts. ? Blank space: An empty field can indicate a missing value. ? Min/Max: It's possible that a number might need to be treated as a missing value if the considered variable has absurd values given the context.

Understanding why data is missing is crucial. Once we know the reason, we can categorize it as MCAR, MAR, or MNAR, enabling us to take actions like omission or imputation. For example, consider collecting data on age and the number of cigarettes smoked: If we don't know why some "Number of Cigarettes" data is missing, it's MCAR. If "Number of Cigarettes" is only missing for young teenagers, it's MAR. If young teenagers who smoke more cigarettes withhold information, it's MNAR

The first step is to assess the impact of the missingness: How Much is Missing? A few scattered missing values such as 1% of the records is very different from missing one-third or more of a feature. While there are no absolute rules for an acceptable level of missingness, I get uncomfortable when I am imputing more than a few percent of the data. Identifying what percentage is missing is very straightforward and you can quickly see where you’re in trouble. Is there a pattern of absence: Are values missing completely at random? Is there a relationship between missingness and other variables? Understanding the why behind missing data helps you choose the right fix.

Missing data can be classified into three main mechanisms: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). MCAR refers to random missingness independent of observed and unobserved data, causing no bias. MAR occurs when missingness depends on observed data but not unobserved data, introducing bias that can be addressed through techniques like multiple imputation. MNAR is non-random missingness related to unobserved data, leading to significant bias that is challenging to handle. Understanding the missing data mechanism helps determine how to handle missing data and assess its impact on analysis.

MCAR, MAR, and MNAR are very distinct in the problems that they present. In the case of MCAR, these missing values can often be estimated by some method because the remainder of the dataset is unbiased. For MAR, these missing values could possibly be estimated, but this requires complete knowledge of the reason why the values are missing and an ability to control for that fact. MNAR is possibly the worst case scenario - values are missing in your dataset because of a systematic bias, therefore, the remainder of your dataset is biased. These missing values are often difficult or impossible to estimate. Often, the difference between MAR and MNAR will be debatable and requires deep understanding of the problem for accurate identification.

There are other ways to classify the missing data. When working with Survey responses, I like to use Domain-specific calculation. For example, in survey data, missing data may be categorized as "refuse to answer," "no response," or "don't know" responses. Understanding that "don't know" and "refusal to answer" are also forms of missing data is crucial. Hence, the reasons behind missingness in the specific domain can help inform appropriate handling strategies. In this particular case, using multiple imputation, model based imputation, assigning different weights techniques look like the best fit to minimize the bias.

Always make sure to look at some samples of your data manually. If you are working with a very large data set, you can't inspect all the missing values but don't ever be lazy and rely on summary statistics only. If you make a habit of always looking manually at some samples, you will, from time to time, discover unexpected things that you wouldn't have learned otherwise.

Using missing/sparse data as a premise to impute with summary statistics may not often be the right approach. Trying to investigate further by reaching out to the upstream teams or business stakeholders always helps. More often than not there is a reason such as a technical glitch in the CRM systems or a business decision which impacted the pattern of the data (availability /sparseness).

One of the largest traps I see people make with missing data is to impute right away. Why? You should only impute your data if the data is missing at random. Thus, it's in your best interest to first understand WHY the data is missing and if there are any patterns to the missingness. From there, you have the following options: 1. Resolve the issue upstream and re-collect the data (most thorough but least feasible). 2. Remove the rows with missing values and note it as a limitation of the analyses or workflow (there are always tradeoffs). 3. Create a dummy variable column of whether a value is missing (e.g. 0/1) to account for this information. 4. Confirm the data is missing at random and impute the data while noting the limitation.

One of the most common ways to impute missing data is Mean Imputation. It’s mainly due to its simplicity but it is by far not the best one. Mean Imputation and other similar methods, such as imputing with 0s, do not just fill datasets with pretty incorrect data, but they also skew the population’s distribution. There are several better ways to impute data and one of them is KNN Imputation. It basically uses a light ML technique to assign the closest neighbors most frequent value. It works by assigning a value closer to similar samples, which tends to be closer to the real-life value on average.

In addressing missing data, it's crucial to select an appropriate method aligned with your data set and analysis goals. Deletion, imputation, and modeling offer distinct approaches, each with its pros and cons. Consider data characteristics, objectives, and assumptions to ensure a robust and accurate analysis. Ultimately, thoughtful handling of missing data can enhance the integrity of your findings and contribute to more reliable and actionable insights.

Cut to the chase, it depends how your data looks like: For Numerical attributes: 1. If outliers are present : Median imputation 2. If outliers are not present: Mean imputation But both the above methods are bogus, however not in every situation, from industry standpoint. Companies make use of more sophisticated techniques to handle missing data like: 1. Mean and median imputation via pandas transform to impute according to each category that avoids introducing bias via newly filled data. 2. There are imputation algorithms like KNN imputer, simple imputer, which predicts the missing value on the basis of stats modelling.

Regression Imputation is also a pretty powerful approach, basically training simple (e.g. linear or logistic regression models) with a subset of the dataset and using it to predict the target columns missing values.

In credit risk models, missing information (especially behavioural or self declared) can tell a story, and have correlation to risk. Identify and code different reasons for the missing if possible, or create a separate WOE for 'missing', analyse behaviour and use in the model. This way missing get a score they deserve rather than average or some imputation based on assumption of randomness.

It depends on the amount of missing values, the variable, and other considerations. One can use WOE, give the a neutral value, or an adverse value. In implementation, handling of missing values should be considered and align with development.

Handling missing data in SAS involves deletion, imputation, or model-based methods. Deletion may lead to bias, while imputation preserves sample size but can introduce bias if misspecified. Model-based methods like Multiple Imputation (MI) are robust but require careful model specification. SAS's handling of special missing values ensures data consistency. Evaluating impact involves considering changes in distribution, variance, and result accuracy. Deletion reduces variance but may skew distributions, whereas imputation and model-based methods preserve data characteristics better. Method choice hinges on missingness nature, analysis goals, and the balance between simplicity and accuracy in SAS.

If it's time series analysis and you are imputing data, be careful not to introduce data leakage (for example using future data to calculate the median which is used to predict the future).

I'd say fancyimpute uses much more advanced methods for data imputation, namely KNN or "IterativeImputer" which applies regression models to predict missing values. the syntax is also pretty mainstream.

You also need to consider the implications on algorithms: There are some tolerant algorithms like decision trees or certain ensemble methods that can handle missing data without major modeling interruptions. Other algorithms such as linear regression have to drop observations when a feature is missing.