How do you handle missing data?

In my experience, before jumping into conclusion that you have a missing data case on you dataset, double check the data queries to make sure that the missing data doesn't come from inaccurate queries. For example, on join table or filtering process part. Keep this in mind especially when you write a very long query syntax. Make sure that your way of accessing the data is absolutely correct and bulletproof.

Something I've discovered as a data analyst is the value of mental and emotional stability while working with datasets. Many errors resulting from missing data can be traced to the data analysts themselves.

Handling missing values in data depends on the Percentage (amount) of missing values in an independent variable. If the missing values are to a greater extent, it is better to delete that column while training the algorithm. There is no point in handling such a large no of missing values and using that parameter in model training. In case the missing values in a column are nearly around 1%, we could delete those rows having missing values. But, in my opinion, this is not the correct approach. This would impact our model training in case the training dataset is very small. There could also be different approaches like imputing values with mean, median, or mode. We could also use an interquartile range to handle the missing values.

A lot of my analytics is based on detecting “anomalistic” behaviors for domains such as fraud detection, tax-evasion, anti-money laundering, financial crimes, etc. So, if there is “missing” data, I need to determine if it’s a result of the collection process (poor interface design, non-mandatory fields, etc) or if it was intentional. Often deception is seen by changing the content provided – different names, switched dates-of-birth, addresses variations. This also includes non-responses which is a “pattern” itself and should always be reviewed. If it turns out it’s a training issue or interface control, then it’s easy to fix – otherwise, it may be an actionable situation.

Understanding of missing data types (MCAR, MAR, MNAR) is vital for model accuracy. For MCAR, mean imputation might suffice, while for MAR, regression techniques may be applied, and for MNAR, a more complex model-specific approach is often required. These targeted methods reduce bias and enhance the integrity of the analysis, reflecting a nuanced approach in handling real-world complexities and ensuring that AI models are both robust and responsive to the underlying data structure.

Adding my perspective here: - Reduction in Sample Size: Missing data reduces the available sample size, potentially reducing the statistical power and precision of the analysis. - Complicates Analysis and Inference: It requires careful consideration and application of appropriate techniques to handle missing values, such as imputation or specialized analysis methods. - Complex Data Modeling: Missing data can pose challenges when building predictive models or conducting data mining tasks. Many algorithms and models require complete data to operate effectively. - Validation and Generalization: Models trained on incomplete data may perform poorly when applied to new data or different contexts.

Let's look at it from another perspective. Missing data, while challenging, can actually open doors for us to dig deeper into the dataset. It's like a puzzle that needs solving, and as data scientists, we have the chance to uncover interesting patterns and reasons behind the missing values.

For example, in a predictive maintenance AI model for aircraft engines, missing data on engine temperature or vibration could lead to incorrect failure predictions. If MAR is present, where missingness might relate to specific flight conditions, merely excluding this data could introduce bias. Specialized imputation techniques would be required to accurately represent the underlying pattern. Failure to address this missing data appropriately could result in inaccurate maintenance schedules, affecting both safety and operational efficiency. Thus, understanding and handling missing data is not just a statistical concern but a practical necessity in AI-driven products.

You should also consider using domain expertise and data engineering approach in identifying missing data. For each features that you will use in your project, you can flag whether the variables could possibly have missing value or not. For example, from business perspective all user data should always have "age" since every users are required to submit their date of birth. If from domain knowledge you are sure that this kind of data shouldn't have missing value, yet you found some missing data, you need to investigate first where does this problem came from. From there, you know how to handle those missing data properly.

From my real world air pollution epidemiology data cleaning experience, the very first step is to get into the data technical documentation to determine what values are assigned to missing data or how the missing values are handled before making the data available to public. Gauging that provides accurate information to strategize further " if those values are vital to study question in context?" If "yes" one can further check summary statistics " how much are we allowing the missing %"? There are several underlying reasons the values for a particular variable is missing i.e. dysfunctional monitors, absence of monitor, or absence of a pollutant/value for that particular day. AHRF files assigns "0" while SEER "unknown" values for missing.

For instance, in the development of a voice recognition AI model, missing data in speech samples could lead to ineffective training. By employing visualizations like heatmaps, you may discover that missing data correlates with specific accents or dialects (MAR). This insight would guide targeted data collection to fill these gaps or the application of specialized imputation techniques to simulate the missing values. Recognizing and understanding this missingness not only enhances the model's accuracy but also its inclusivity, ensuring that the end product is attuned to a more diverse user base. It's a demonstration of how data understanding translates into product excellence.

Handling missing data is one of the important decisions to make in Data Science projects. Subject Matter Experts use case type and volumetric of missing data, and nature are decision influencers and factors here. Missing at Random to Missing not at Random requires varied approaches. We can follow certain rules of thumb, like missing 30%, but at random with recognizable patterns, and fill it with imputing techniques. Some types of missing data (continuous) can be imputed with regression techniques. The successful recipe should be hand-crafted by the Data Scientist in consulting with an SME.

To identify the nature of missing data, we can do an exploratory data analysis on it using missing-data heatmaps or proportion analysis to identify specific trends in missing values. We can do an imputation impact analysis to evaluate the effects of missing data on the results of your data analysis project. Approach used to handle missing data also depends on the domain to which the data belongs. E.g., in medical data, we can sometimes impute the missing lab results based on the available medical history of the same patient. We also need to be cautious of regulatory and data privacy needs while dealing with missing data. Specialized techniques like pattern mixture models and Bayesian modelling can be more effective for sensitive data.

Missing value is a missing value period. After identifying underlying reasoning for missing next question is "is it a confounder in your statistical model that affects your study context causal pathway?" If "yes", and its a categorical variable, replace those missing values into "unknown" category to be able to use the variable in the data analysis. I came across several missing values in daily air pollution data, which differ spatiotemporally so imputing or extra/interpolating seemed inaccurate. Created air pollutant variables with "upto %missing" i.e. "upto 50%", "33.33%", "25%", "20%" and reported in results section of study for transparency purpose about how much% missing values were allowed for each air pollutant.

In a real-world scenario such as optimizing a recommendation engine for an e-commerce platform, the choice among deleting, imputing, or ignoring missing data can have substantial impacts. Deleting user browsing data might strip away nuances in user preferences. Imputation, while preserving the dataset's structure, might introduce bias if not done carefully, potentially skewing recommendations. Ignoring missing data using advanced techniques like matrix factorization might be ideal but could be computationally intense. The decision thus demands a careful understanding of the trade-offs involved, aligning with the product's goals and the nature of the missing data.

If I'm taking data from a production system to analyze it, if there are a lot of "missings" in a variable we want to use in analysis, I worry that the item was not measured the way we thought it was - or it shouldn't be missing so much. In a production environment, this can be a signal you might need to take a different approach to operationalizing that variable for your analysis. Maybe you need to use a different variable that means something similar but is more complete.

When to delete a feature that has missing values? The wisdom says, when the percentage of missing values exceeds a certain threshold (i.e. 80%). However, this is still a loss of information. Sometimes, it's better to impute the missing values with some constant (i.e. 0 or -1) and keep the remaining non-null values intact. When used with tree-based algorithms, the algorithm can use this extra piece of information and ignore the constant value. However, you should always experiment and compare since this is a just an observation I had without rigorous verification.

How to prevent missing data: Deleting Rows with missing values. Impute missing values for continuous variable. Impute missing values for categorical variables. Using Algorithms that support missing values. Prediction of missing values.

Model-Based Imputation: Use statistical models to predict missing values based on other variables in the dataset. Regression imputation and k-nearest neighbors (KNN) imputation are examples of model-based approaches. Multiple Imputation: Generate multiple imputed datasets using statistical techniques and combine the results to address uncertainty in imputed values.

Steps to take when Handling Missing Data: 1. Inspect the data to understand the extent and pattern of missing values. 2. Understand the reasons for missing data and use domain knowledge to determine if it's safe to ignore or impute the values. 3. Choose an imputation method based on the type of data (mean, median, regression, or multiple imputation) and evaluate its performance. 4. Account for uncertainty in the imputed values and avoid overfitting. 5. Perform sensitivity analysis to assess the impact of missing data on the findings. 6. Continuously monitor the performance of the imputation method throughout the project lifecycle. Note: Document all steps taken to handle missing data for collaboration, replication, and auditing purposes.

1. Validation and Sensitivity Analysis: Ensuring no bias is introduced through handling, and testing the impact of different methods. 2. Documentation and Communication: Recording the handling process for transparency and collaboration. 3. Regular Monitoring: Continuously checking for new instances of missingness, enabling timely action. 4. Creating Robust Processes: Implementing standards to minimize missing data during collection. For example, in managing inventory for a supply chain AI product, these steps contribute to precise decisions and operational efficiency. Handling missing data with care is pivotal to product success.