登录查看更多内容

What are the most common techniques to detect outliers?

由人工智能和领英社区提供技术支持

Outliers are data points that deviate significantly from the rest of the distribution, and can affect the accuracy and performance of machine learning models. Therefore, detecting and handling outliers is an important step in data cleaning and preprocessing. In this article, you will learn about some of the most common techniques to detect outliers, and their advantages and disadvantages.

此文章中的业界达人

由社区从 6 条内容中精选。了解更多

1 Z-score method

The Z-score method is based on the assumption that the data follows a normal distribution, and calculates the standard deviation of each data point from the mean. A data point is considered an outlier if its Z-score is above a certain threshold, typically 3 or -3. This method is simple and fast, but it can be sensitive to the presence of outliers, and may not work well for skewed or non-normal distributions.

添加您的观点

2 Interquartile range method

The interquartile range (IQR) method is based on the concept of quartiles, which divide the data into four equal parts. The IQR is the difference between the third and the first quartile, and represents the spread of the middle 50% of the data. A data point is considered an outlier if it is below Q1 - 1.5*IQR or above Q3 + 1.5*IQR, where Q1 and Q3 are the first and third quartiles, respectively. This method is more robust than the Z-score method, as it is not affected by the mean and standard deviation, but it can also miss some outliers, especially if the data is not symmetric.

添加您的观点

3 Boxplot method

The boxplot method is a graphical way of using the IQR method to detect outliers. A boxplot is a plot that shows the median, the first and third quartiles, and the minimum and maximum values of the data, excluding the outliers. The outliers are represented by dots or circles outside the boxplot. This method is useful for visualizing the distribution and variability of the data, and identifying potential outliers, but it can also be subjective and depend on the choice of the whiskers, which are the lines that extend from the box to the minimum and maximum values.

添加您的观点

4 Isolation forest method

The isolation forest method is a machine learning algorithm that detects outliers by isolating them from the normal data points. The algorithm randomly splits the data into smaller subsets using binary trees, and assigns an anomaly score to each data point based on how many splits are required to isolate it. The fewer splits, the higher the score, and the more likely the data point is an outlier. This method is effective and scalable for high-dimensional and large datasets, but it can also be sensitive to the parameters and the randomization.

添加您的观点

Patrick Nicolas

Director Data Engineering @ aidéo technologies |software & data engineering, operations, and machine learning.
举报内容
In case these techniques were not mentioned... - Single class Support Vector Classifier for which the origin is used as second class to compute the margin - Manifold learning (as an extension of PCA for non-linearity) such as Laplace eigenmaps

已翻译

赞

5 DBSCAN method

The DBSCAN method is a clustering algorithm that detects outliers by grouping data points based on their density and proximity. The algorithm defines a core point as a data point that has at least a minimum number of other data points within a given radius, and an edge point as a data point that is within the radius of a core point, but not a core point itself. A data point that is neither a core point nor an edge point is considered an outlier. This method is flexible and can handle arbitrary shapes and sizes of clusters, but it can also be challenging to choose the optimal parameters and deal with varying densities.

添加您的观点

Can Koyuncu

Machine Learning Software Engineer
举报内容
The DBSCAN method is a density based clustering algorithm and can be used to detect outliers. The pros and cons are: Pros: - Doesn't require specifying the number of clusters. - Can find arbitrarily shaped clusters. - Robust to noise and outliers. - Handles clusters of varying densities effectively. Cons: - Sensitive to proximity parameter. - May not work well for high-dimensional data. - Computationally intensive.

已翻译

赞

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Sid Roy

Senior Research Scientist @ Protopia AI | PhD, Deep Learning/LLMs
举报内容
AutoEncoder for Anomaly Detection AutoEncoder is a Deep Neural Network architecture that has an encoder and a decoder portion. The encoder portion compresses the input data to a lower dimension and the decoder layers reconstruct the data back to the input data dimension. The reconstruction error that is used to train the Autoencoder can be used to detect the anomalies. First, a threshold can be computed by calculating the mean and standard deviation of the reconstruction errors. Then, points having reconstruction errors beyond the set threshold can be treated as anomalies.

已翻译

赞
Josh Johnson

Data Sciencist with expertise in Statistical Analysis, Machine Learning, and Generative AI
(已编辑)
举报内容
You can use PCA to detect anomalies, too. Transform the data into a small number of PCs, then do an inverse transform and compare the original data with the transformed/inverse transformed data and note which data points have the largest difference (error).

已翻译

赞
Ranganath Venkataraman

Digital Transformation through AI and ML | Decarbonization in Energy | Consulting Director
举报内容
The box and whisker plots are a popular choice not only because they serve the purpose of identifying outliers but they also act as useful visuals for presentations and reports.

已翻译

赞
Utkarsh Wadhwa

???? AI Engineer?? Generative AI ?? Machine Learning ?? MLOps ?? Deep Learning ?? Natural Language Processing ??? Computer Vision ?? Ex-Cognizant, upGrad ???? IIIT-B Alumni
举报内容
Let me list other methods which are used as well: Local Outlier Factor (LOF): LOF measures the local density deviation of a data point with respect to its neighbors. Points with a significantly lower density are considered outliers. Mahalanobis Distance: This method considers the correlation between variables and identifies outliers based on their distance from the center of the data distribution. Tukey's Fences: Similar to the IQR method, it defines lower and upper boundaries beyond which data points are considered outliers. Machine Learning Models: Some machine learning algorithms can be trained to identify outliers, such as one-class SVM and autoencoders.

已翻译

赞

Machine Learning

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

What are the most common techniques to detect outliers?

1

2

3

4

5

6

1 Z-score method

2 Interquartile range method

3 Boxplot method

4 Isolation forest method

5 DBSCAN method

6 Here’s what else to consider

Machine Learning

给文章评分

感谢您的反馈

更多Machine Learning相关文章

更多相关阅读内容

What are the most common techniques to detect outliers?

1

2

3

4

5

6

1 Z-score method

2 Interquartile range method

3 Boxplot method

4 Isolation forest method

5 DBSCAN method

6 Here’s what else to consider

Machine Learning

给文章评分

感谢您的反馈

查看其他技能