How do you handle data outliers?

You can of course apply the mahalanobis distance and identify the outliers. Another option when coping with large multivariate data is applying PCA, ICA, t-SNE or UMAP to overcome the high dimensionality and use the new introduced latent variables or components as new dimensions to eliminate outliers by an iqr outlier elimination or x standard deviation or median absolute deviations away from the mean/median for each of the latent variables

The key here is to identify outliers and report on them, not necessarily eliminate them as this may skew results. I recently completed a study on train derailments of which I plotted various longitude and latitude points within the continental United States. Some of the plotted points ended up scattered throughout the Atlantic Ocean. Despite this, I chose to retain the points on the map to demonstrate that such discrepancies were relatively minor in the context of the larger picture. By showcasing these outliers, the study underscored the importance of acknowledging and understanding the occasional inaccuracies in incident reports while maintaining focus on the overall trends and patterns in the data.

As with all statistical techniques and analyzes, there are measures of statistical 'significance' . However, it's so important that we also consider the contextual implications of outliers rather than removing them because they are statistically distanced from the rest of the points in the distribution. How you deal with outliers must be determined by the context of the research question. Outliers, as long as they are not a result of methodological mistakes, might be some of the most important points in your data set. It very much depends on your context and what is at stake!

In some cases, outliers might not be as obvious, especially when there are data collection issues. For example, this is pretty common in data pertaining to movement and health. In such cases, there may be multiple points that become outliers once filtered for practical constraints like the limits of the human body or physics.

Identifying and fully analyzing outliers requires focusing on or filtering for each factor in the data one at a time. There may be general outliers when looking at the data as a whole or as an aggregate set. But there may be specific outliers within a particular filter view of the data. For example, you may know the outliers in the set of top net wealth individuals in the world. But do you know the outliers in the set of top net wealth individuals in your local region? Do you know the outliers in the set of top net wealth individuals under 30 years old in your local region? Filtering for specific factors within the data can offer you insight into the outliers for different viewpoints.

A general predictive model typically has an intense optimization focus on learning the global properties of the distribution, by discarding outliers from the dataset during its training stage. In this context, such models fail to capture properties of outliers observed in the tail of distributions. In order to address this type of issues, here is an alternative case. “Tail Generative Adversarial Network (Tail-GAN) ”, a learning algorithm based on Generative Adversarial Networks, focuses only on local optimization in the tail of distribution. It optimizes its predictive power on specific properties of the tail of distribution. Combining with conventional predictive model, we can separately analyze the non-outlier dist. and the outlier dist.

Some techniques that help to manage outlier bias include: - Examining the entire range of distribution of values - Using median instead of average (measure of central tendency) - Finding and investigate outliers - Removing outliers, this is a fast solution - Replacing values where, any outlier valued as less than a minimum accepted value is replaced with a the minimum value; or, any outlier greater than the maximum accepted value is replaced with the maximum value - Capping by setting a limit; ie. above a particular value or less than a specified value, all values will be considered as outliers - Binning involves making groups, where we put outliers into a group and force them to behave in the same way that other points in the group behave

Outliers can be represented in various ways. While not all outliers are directly pertinent to the research question it is important to consider the ways in which the data may benefit from a contextual acknowledgment, especially in regards to the relative research and broader projects which may be at the root of the research study.

Regardless of your decision about how to handle outliers, transparency is always key. Explain the rationale and methodology and allow others to decide whether they agree or disagree with your choices.

Outliers can provide insights into the underlying distribution of the data. They may indicate the presence of subpopulations or rare occurrences that are relevant to the study. Understanding the nature of the outliers can lead to a more comprehensive and nuanced analysis, allowing researchers to consider the impact of extreme values on their results.