How can you handle missing context and background data in a dataset?

When going through the data for the project. Understanding the data and the importance of each column is very important. Checking for null values and outliers is very crucial as it can skew the entire visualization. Exploratory data analysis can be used to identify the outliers and inconsistencies in the data. As per the importance of the data in the visualization, these values can be highlighted or ignored.

Understanding the data you're working with is critical when dealing with missing data. Most missing data appears as a NULL value, which is easy to identify, but sometimes the data is filled with a dummy value that makes no sense within the context of the feature. For example, a chemical with density of 0.0, or a zip code listed as 0. This underscores the importance of exploratory analysis - identifying where the missing values are and how they are represented. Do the values make sense within the range? Do the missing values impact the target metric? If the missing values have no impact, it's possible to remove the feature altogether. If impactful, we may need to use external resources to fill what we can.

Missing context and background data can be a significant challenge in working with datasets, as it can lead to biased results and inaccurate conclusions. Here are some approaches you can use to handle missing data: -Mean/Median/Mode Imputation -K-Nearest Neighbors (KNN) Imputation -Synthetic Data Generation -Create new features -Choose models that are robust to missing data -Use ensemble methods -Incorporate domain expertise -Use explainable AI techniques -Be transparent about missing data -Evaluate the impact of missing data -Consider the limitations of your approach you can develop effective strategies for handling missing context and background data in your datasets and improve the accuracy and reliability of your results

Estimating the impact of missing data is similar to searching for a small object in a darkened room. This can be an extremely challenging task. If the problem is a well-defined one, it is similar to having a flashlight- there are certain areas in the room we know to look into. For instance, if we are training a generic dog vs cat model and our dataset consists of very few dogs and lots of cats, we are prompted to get more images of dogs to balance out the data. On the other hand, if it is purely exploratory data with very little metadata or documentation, it may be better to train preliminary models before making guesstimates which may ultimately prove ineffective.

Understanding the features of the data are critical in assessing the impact of the missing data. Not all data are equal. If we haphazardly remove records with null values, we may be introducing bias into other analyses that we are unaware of. This is where data visualizations are paramount in the exploratory analysis. We can see how the relationships of the other features change when we remove the subset of data missing certain values. The impact of removing a few rows with missing data is much less than the impact of removing a systematic exclusion of data. The use of statistics to determine if there is a significant change in the distributions that shouldn't be impacted by the missing feature can help in this decision.

Na minha experiência, para analisar o impacto de dados faltantes deve incluir os seguintes itens: 1 - Analisar a dimens?o da completude da sua análise: qual é o % faltante; 2 - identificar se existe algum padr?o de “dados faltantes”: como por exemplo: hora, data, sistema que apresente algum tipo de falha sistêmica; 3 - Impacto da análise: Se os dados faltantes distorcem a análise, ou impulsionam tomadas de decis?es erradas; 4 - Transparência: comunicar os usuários que foi identificado dados faltantes e traduzir o que isso pode impactar em suas respectivas análises (aumentando o crédito de uma cultura baseada em dados).

Using data visualization during the exploratory analysis can really show on a gross scale whether there is a difference between ignoring or removing the data. When there is a substantial impact, the analyst needs to decide whether the features analyzed are changing the meaning of the results, which comes back to really understanding the features that are being measured. Missing measurements are going to have a different impact than missing identifiers, such as an address or zip code. Imputing data is very much impacted by this. Imputing data for a measurement can be done statistically, using the mean or median, but should be carefully thought through based on whether the other data in that row are similar to the other data.

When working on a project with pretty poor data quality, it was clear that certain metrics were collected from a device, and these had high fidelity. Other aspects were filled in by a person, and often on different days, leaving variations in spellings, missing fields, and impossible values (like a temperature of -99 degrees in Philadelphia) A lot of these values could be found based on public data on addresses, weather conditions, and naming. In a different study, different chemical naming conventions and units were used from different facilities, but there were missing values such as molecular weight and density from one facility that were provided by others. In this case imputing the data was an easy decision.