Missing data is a common and inevitable problem in data science. It can affect the quality, validity, and reliability of your analysis and results. In this article, you will learn some basic concepts and methods to deal with missing data in your data science projects.
Missing data indicates that some values in a data set are not recorded or available. This can be classified into three types: Missing completely at random (MCAR), Missing at random (MAR), and Missing not at random (MNAR). Each type of missing data has a different effect on how it can be handled and what assumptions need to be made. MCAR means the missing values are unrelated to any other variable or the outcome of interest, MAR means the missing values depend on some other observed variables, but not on the outcome of interest, and MNAR means the missing values depend on the outcome of interest or some other unobserved variables.
Adding my thoughts here: While missing data can be fundamentally classified into MCAR, MAR, and MNAR, one has to recognize that in the real world, a dataset might involve a mix of these mechanisms, making it challenging to distinguish between them. Data could be missing for a variety of reasons, such as data collection errors, participant non-response, or system malfunctions. Depending on the amount of data missing, statisticians and data practitioners often rely on different statistical and exploratory data analysis techniques to make informed assumptions. From a non-data science perspective, when analyzed thoroughly, missing data often helps us identify points of failure in our systems
Tackling the Missing Data Challenge: Data Imputation: Employing innovative imputation approaches to fill the void left by missing data and avoid skewed interpretations. Assumption Management: Making informed assumptions based on the type of missingness to avoid misleading conclusions. Robust Analysis: Ensuring that missing data does not undermine the reliability and credibility of our analytical endeavors.
More often than not, one can expect to encounter missing data when working with real datasets. It is important to identify the cause of missingness in the data. Key questions to investigate could be: Is the data missing randomly, or does it follow a pattern? Basis the diagnosis, one can use simple methods like Mean/Median/Mode to replace the missing data, depending upon the request or ask.
Handling missing data is a nuanced task that requires understanding the nature of the missingness (MCAR, MAR, MNAR). In my practice, I emphasise proper diagnosis first. For MCAR, simple imputation or deletion might suffice. For MAR, techniques like multiple imputation can be more appropriate. MNAR is trickier and may require domain knowledge. Always, the choice of method must align with the data's underlying structure and the analysis's goals. Transparency about the approach taken is crucial for the validity and reliability of the results.
Well, before anything, I believe we have to step back from the technical approaches and ask: why this data is missing? Did we talk with the data owners and creators why we got missing data? Is this data missing because of the collection process, the data pipeline, or the data is not possible to be collected? Did you know that you had missing data before reaching out to the Data Scientists? Only having the context at hand is that I believe we can start opening our toolbox and consider the next questions. Restricting the discussion on time-series data, is there any temporal correlation in the data columns? Do the phenomena being studied present seasonal or cyclical effects? With that, we can then propose and test interpolation, etc.
Missing data can introduce bias, reduce statistical power, and complicate your analysis, potentially distorting the distribution and relationship of your variables. It can also reduce the sample size and representativeness of your data, affect the estimation and inference of your models and parameters, and increase the uncertainty and error of your predictions and conclusions. Consequently, you must identify, understand, and address the missing data in your data set before proceeding with your analysis.
Dealing with outliers is of utmost importance because they can significantly skew data and reduce the reliability of analyses. Outliers can exert disproportionate influence on statistical measures, such as the mean and standard deviation, leading to distorted distributions and misleading interpretations of the data. These extreme data points can bias parameter estimates, compromising the accuracy and validity of models. If not properly handled, outliers can disrupt the assumptions underlying many statistical methods, affecting the robustness and generalizability of the results. Therefore, identifying and appropriately addressing outliers is crucial to ensure the data's integrity and maintain the reliability of analyses.
The purpose of any analytical efforts is to bring out information from data so that the decision makers can make informed decisions. Depending on the situation and use case, these decisions might cost a company millions of dollars. Hence to maintain the core values, one has to ensure that we are not ignoring the missing information. While it is hard and impractical in practice to pen down a reason for every missing data point, considering the impact of missing data, performing a root cause analysis will save an organization a lot of time, energy and money in the future.
While I agree that missing data must be addressed, I disagree with the perspective on dealing with outliers. In security, outliers can provide valuable information for fraud detection and anomaly identification. Leveraging robust outlier detection methods, such as clustering-based approaches, distance-based techniques, or anomaly detection algorithms like Isolation Forest or One-Class SVM, can strengthen security measures and improve fraud detection systems. Understanding the unique value outliers and anomalies bring to security is essential for informed decision-making and reliable security analyses.
Missing data impacts can vary - Missing Completely At Random data *may* not introduce bias but can reduce statistical power, while Missing Not At Random data can introduce significant bias (as the "missingness" is related to the unobserved data itself). Addressing the ongoing conversation about outliers, it's important to distinguish them from missing data. I agree that outliers, especially in security, can provide valuable insights for fraud detection. And like missing data, they require appropriate handling based on the context. They can also be due to errors, which can skew data/reduce the reliability of analyses. Effective data science hinges on understanding your data's context and complexities, not on a one-size-fits-all approach.
Addressing missing data involves strategies like excluding incomplete records and imputation using mean, median, or regression (with possible bias). Temporal data may use interpolation. Advanced machine learning predicts missing values, leveraging more features. Augmenting with indicators is an option. Optimal strategy depends on analysis, impact assessment, and goals. Documenting ensures transparency.
The first step to handle missing data is to identify how much is missing and where it is in your data set. Exploratory data analysis, data quality checks, and metadata analysis are all useful tools and techniques to do this. Summary statistics, visualizations, and descriptive methods can help detect any missing values or patterns. Validation rules, integrity constraints, and anomaly detection can verify the accuracy, completeness, and consistency of your data. Additionally, metadata like data dictionaries, documentation, and provenance can provide the source, meaning, and context of your data. After identifying missing data, you can assess its impact and decide how to handle it.
In addition to the obvious and mentioned techniques for identifying missing data (which are great 1st steps), it's worth highlighting the role of predictive modeling in this process. Shameless plug for ML here, but algorithms can also be trained to predict the likelihood of "missingness" in data based on other available features. This approach can be particularly useful in large datasets where manual inspection is just not feasible. On top of that, it can help in understanding the underlying mechanisms of why and how it's missing, which is crucial for choosing the most appropriate strategy for handling missing data (and will likely come in handy as you need to make further decisions for your analysis of this dataset).
In my experience, identifying missing data and just checking data dictionaries can be missleading. More often than not I learnt that reaching the SME specific for this data source has been very valuable. Not everything is easily documented, especially when we have human generated data (for example, in a CRM). The complexity in real world examples for entering data into a system, also makes almost impossible to understand all subtleties with just documentation and dictionaries.
Missing data can be identified through visual inspection, summary statistics, and data exploration techniques. Common indicators of missing data include empty cells, placeholder values, or domain-specific codes that denote missing values. Especially helpfull in real world scenarios is do pattern identification and look for interuptions.
When dealing with missing data, the type, amount, and pattern of the missing values, as well as the goal of your analysis, should be taken into account. Deletion is a simple and fast method that involves removing rows or columns that contain missing values. However, it can reduce your sample size and introduce bias if the missing data is not MCAR. Imputation is a more complex and flexible method that replaces the missing values with estimated values based on other available information. Yet, it can add noise and uncertainty if the imputation is not accurate or appropriate. Lastly, model-based approaches use statistical models or machine learning algorithms to account for or ignore the missing values in your data set. This method is more advanced and robust. However, it may require more computational resources and assumptions. Ultimately, you should choose the method that best suits your data and analysis, and evaluate its performance and validity.
There are several ways to handle missing values. One way is if the missing values doesn't affect the data, they can be deleted. Another way is to imput ways such as mean or average of the values from the corresponding values. Another best way, is to impute values by modeling techniques like regression that predict what can be the best value.
This problem is common in time series datasets, where there are snippets of the series with missing information that disables the understanding of the events of that time frame. This occurs in the data collection phase, where the sensors and devices have connection problems and don’t gather the information in the required frequency. It is important to handle these missing pieces since the amount and quality of data is crucial for any insights and model inference. To handle this, we can inference the trend of the curve from the other previous information or just use the mean/median of the curve to represent the dataset.
From a modelling/reporting standpoint, we want to retain as much data as possible. We first look to understand why the data is missing before proceeding to think about how to handle it. Once we have the understanding, we have the following options - Remove as it is due to data errors. This is fine if the data is small but when it starts to account for a significant portion of the data, this isn't ideal - Default it to the most common value (mode) for the specified column provided it doesn't change any metric materially of the most common value in the column - Classify it as its own category as the volume is sufficient and you expect similar issues to continue. - Enrich the data using other data sources to fill in the missing data
Sometimes people are hesitant to drop data points with missing data, but it is sometimes the correct thing to do. Ideally you would want to preserve as many features as possible, but there are times when your dataset is either large enough or you lack the industry experience/context to be able to impute missing data. Don't be afraid to drop records with missing data for the first iteration, it can help you get your model/analysis up and running quicker and help save valuable time and resources that would have gone to imputing missing data.
The choice of method depends on the nature and extent of missingness, as well as the specific goals of the analysis. Most important: never modify the raw data. Always work on a copy of the data set. From an academic point of view missing data and the handling method needs to be indicated. I'd recommend to build a validated process.
To prevent missing data, you should focus on improving your data collection and management processes. This means using clear and consistent definitions, formats, and protocols to collect, store, and transfer your data. You should also enhance your data quality and governance practices by implementing standards, policies, and procedures to ensure the reliability, security, and usability of your data. Lastly, it is important to communicate and collaborate with your data stakeholders in order to understand their needs, expectations, and feedback. By taking these steps to prevent missing data, you can improve the efficiency, effectiveness, and credibility of your data science projects.
Preventing missing data is pivotal in maintaining the integrity of data science projects. In my experience, a proactive approach involves collaboration between data collectors, analysts, and stakeholders. Clear guidelines, robust training, and real-time validation during data collection can minimise errors. Implementing automated checks and balances, along with rigorous data governance policies, ensures consistency. Regular audits and feedback loops with stakeholders help in early detection of issues. By fostering a culture that prioritises data quality and open communication, you create a resilient framework that significantly reduces the occurrence of missing data.
Implementing validation checks during data entry helps identify and correct errors or missing values in real-time. In my eyes it's the most pragmatic way.
Preventing missing data is crucial for data integrity and accurate analysis. Employ standardized data collection methods and clear protocols to minimize errors. Implement data quality and governance practices to maintain reliability and security. Foster open communication with stakeholders to better understand their needs. By prioritizing these measures, you enhance the overall effectiveness of your data science projects and ensure sound decision-making based on reliable data.
In our case, we believe so much in data integrity that we have several data steward positions that proactively check the reliability of data. In addition, we frequently monitor the data quality in several degrees of depth depending on the associated risk. This, together with clear guidelines, helps us creating a quality culture.
As a leader, guide your team in handling missing data, outliers strategically, and purposefully designing systems. Address missing data with appropriate methods like imputation or statistical models. Utilize robust outlier detection and analysis methods for valuable insights in security and fraud detection. Implement data governance and cleaning to ensure high-quality, reliable data. Elevate efficiency and reliability for successful outcomes.
It is important to take into account the human dimension for missing data. Of course sometimes there are systems and technologies that just fails, but much of data is human generated. There is the issue of what to incentivize to prevent both missing but also false data. But also the issue that humans tend to not make an effort when they believe something won't be used. So we need also to be careful on what are we asking to introduce and why, so we can later on reinforce the importance in business terms, and not just that quality is important.