登录查看更多内容

How can you use machine learning to identify outliers in your data?

由人工智能和领英社区提供技术支持

Outliers are data points that deviate significantly from the normal distribution of a dataset. They can be caused by errors, anomalies, or rare events, and they can affect the accuracy and reliability of your business intelligence (BI) analysis. Machine learning (ML) is a branch of artificial intelligence that uses algorithms and data to learn from patterns and make predictions. You can use ML to identify outliers in your data and handle them appropriately. Here are some ways to do that.

此文章中的业界达人

由社区从 3 条内容中精选。了解更多

Dr.Sumathi s

Artificial Intelligence Researcher-Specializing in NLP, GenAI, LLMs, and RAI
Colin Hammond

Creator of AI Software Requirements Analysis Tools - automated estimation, QA and insight.

1 Use unsupervised learning

Unsupervised learning is a type of ML that does not require labeled data or predefined rules. It is used to discover hidden structures and patterns in your data, such as clusters, associations, or dimensions. You can also use unsupervised learning to identify outliers by finding data points that do not belong to any cluster or have low similarity to other data points. Common unsupervised learning methods for outlier detection include K-means clustering, Isolation forest, and Local Outlier Factor (LOF). K-means clustering partitions your data into k groups based on the distance to the nearest centroid. Isolation forest isolates data points by randomly splitting the feature space into subspaces. Local Outlier Factor (LOF) measures the local density of data points by comparing the distance to their nearest neighbors. By using these methods, you can identify outliers as data points that are far away from any centroid, have a high within-cluster distance, are isolated with fewer splits, or have a low local density or a high LOF score.

添加您的观点

Dr.Sumathi s

Artificial Intelligence Researcher-Specializing in NLP, GenAI, LLMs, and RAI
举报内容
another way to find outlier is Z-Score, Modified Z-Score, and Box Plot (IQR Method) , DBSCAN (Density-Based Spatial Clustering of Applications with Noise), SVM

已翻译

赞
Azizul Hakim

SWE @ AoE | Python | Machine Learning | Angular | NodeJS
举报内容
Another easy way to detect outliers through unsupervised learning is to use standard deviation. We can calculate the Z-score, which measures how many standard deviations a data point is away from the mean. Data points with high absolute Z-scores are considered outliers.

已翻译

赞

2 Use supervised learning

Supervised learning is a type of ML that requires labeled data and predefined rules. Through the input-output relationship, it can learn and make predictions based on new data. You can use supervised learning to identify outliers by treating it as a classification or regression problem. Three common supervised learning methods for outlier detection are logistic regression, support vector machine (SVM), and neural network. Logistic regression models the probability of a binary outcome, such as outlier or not, based on the linear combination of the features. Support vector machine finds the optimal hyperplane that separates the data into two classes, such as outlier or not, with the maximum margin. Neural network consists of layers of interconnected nodes that process the features and output a prediction - outlier or not. These methods can help you identify outliers as data points that have a low probability of belonging to the normal class or have a high residual error, lie on the wrong side of the hyperplane or have a large distance to the hyperplane, or have a low prediction score or have a high loss function.

添加您的观点

Colin Hammond

Creator of AI Software Requirements Analysis Tools - automated estimation, QA and insight.
举报内容
We use fixed models to detect functional intent in software requirements. We then present the results graphically and in a tabular form to expose outliers, inconsistencies, duplicates and omissions. One of the many benefits is the assurance of it being responsible AI.

已翻译

赞

3 Use semi-supervised learning

Semi-supervised learning is a type of ML that combines labeled and unlabeled data, allowing the model to benefit from both types of information. This technique can be used to detect outliers by training the model on labeled data and testing it on unlabeled data. Popular semi-supervised learning methods for outlier detection include One-class SVM, Autoencoder, and Label Propagation. One-class SVM identifies outliers as data points outside the boundary or with a negative output. Autoencoder looks for outliers with high reconstruction error or low similarity to the input data. Label propagation assigns labels to unlabeled data based on similarity to labeled data, with outliers having a low similarity or conflicting label.

添加您的观点

4 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Business Intelligence

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can you use machine learning to identify outliers in your data?

1

2

3

4

1 Use unsupervised learning

2 Use supervised learning

3 Use semi-supervised learning

4 Here’s what else to consider

Business Intelligence

给文章评分

感谢您的反馈

更多Business Intelligence相关文章

更多相关阅读内容

How can you use machine learning to identify outliers in your data?

1

2

3

4

1 Use unsupervised learning

2 Use supervised learning

3 Use semi-supervised learning

4 Here’s what else to consider

Business Intelligence

给文章评分

感谢您的反馈

查看其他技能