登录查看更多内容

How do you detect and handle outliers in regression analysis?

由人工智能和领英社区提供技术支持

Outliers are data points that deviate significantly from the general pattern of the rest of the data. They can affect the accuracy and reliability of regression analysis, a common technique for exploring the relationship between variables in business intelligence (BI). In this article, you will learn how to detect and handle outliers in regression analysis using some simple methods and tools.

此文章中的业界达人

由社区从 5 条内容中精选。了解更多

Lauren Ainsley Haines

Public Policy | Data Science | Business Intelligence
Shubh Bajpai

Data Scientist @KPMG | Computer Science Engineer @BITS Pilani
OJ Taiwo MBA, MCIM

Business Analyst | Marketing Data Analyst | Data Scientist | IT & Digital Transformation | Driving Growth with AI…

1 Why outliers matter

Outliers can have different causes, such as measurement errors, data entry mistakes, natural variability, or unusual events. Depending on the type and extent of outliers, they can distort the regression results in various ways, such as inflating or deflating the slope, intercept, or coefficient of determination, or increasing the standard error or variance. This can lead to misleading or erroneous conclusions about the strength, direction, or significance of the relationship between the variables.

添加您的观点

Shubh Bajpai

Data Scientist @KPMG | Computer Science Engineer @BITS Pilani
举报内容
Outliers in data are crucial elements that need careful consideration, especially in regression analysis.Think of this example Imagine running a coffee shop with steady, predictable sales. One day, you hold a special promotion, and your sales skyrocket. A few weeks later, you analyze your sales data to plan your budget for the next month. If you don't account for this outlier, you might assume that your sales have permanently increased and allocate your budget based on this assumption. This could lead to overspending or overstocking, expecting sales levels that were actually just a one-time spike. Just like this scenario, outliers in regression analysis can give a distorted view if not properly identified and handled.

已翻译

赞
OJ Taiwo MBA, MCIM

Business Analyst | Marketing Data Analyst | Data Scientist | IT & Digital Transformation | Driving Growth with AI, Machine Learning & Data-Driven Solutions
举报内容
There are many ways to detect and deal with outliers, especially in regression analysis. Identification method: -Visualize using a scatter plot or box plot - Implement statistical methods such as cook distance, residual method etc After identification, managing outliers can be done depending on the quantity and nature of the data. Different ways this can be done is either to remove them, transform or implementing a robust regression technique. Using regression has to be sensitive to outliers and data quality must be put into consideration in these processes.

已翻译

赞
Kirti Kureel

Data Science & analytics COE @ CNH Industrial | LinkedIn Top Voices | MTech | Google, Microsoft and IBM Certified Data Professional |BI | Mentor| Azure| Power BI| Qliksense| Qlik | Python| R| Agile| Project Management
举报内容
Outliers are the set of datapoints which fall out of the data patterns which we identify during our analysis. The reason why we want to eliminate them is because they might skew the results of or regression analytics. There might be many reasons why they occur at the first place like manual error, natural reasons. They must be detected by using any of the following methods. Statistical analysis like cooks distance measurement, standardized residuals or graphical representation using scatter plot, box and whiskers plot using Data visualization tools like Qlik and power BI. The choice of methods and techniques to handle, may be depending upon the situation and there is no one size fits all solution.

已翻译

赞
Ravi Palankar

Human, Interested in Humanity,??Top Leadership Voice, VP - CreateNewThings
举报内容
No data is perfect, we have to clean the data and make sure the data falls in category when we are trying to build a model. The training data is very important. For datasets to meet quality standards, outliers should be addressed to ensure the accuracy, reliability, and integrity of the data. Outliers can lead to overfitting, where the model fits the training data too closely and performs poorly on new data. handling outliers in regression analysis is essential for building reliable, accurate, and interpretable models that adhere to statistical assumptions. It improves the model's performance, predictive capabilities, and robustness, ultimately contributing to more informed decision-making.

已翻译

赞

2 How to detect outliers

Regression analysis has several methods for detecting outliers, such as visual inspection, calculating standardized residuals, and measuring Cook's distance. Visual inspection involves using scatter plots or box plots to identify data points that are far away from the main cluster or the expected range of values. Standardized residuals are the differences between the observed and predicted values divided by the standard deviation of the residuals, with any value larger than 3 or smaller than -3 considered an outlier. Cook's distance is a statistic indicating how much each data point influences the regression coefficients, with any value larger than 1 or 4 times the mean Cook's distance considered an outlier.

添加您的观点

Lauren Ainsley Haines

Public Policy | Data Science | Business Intelligence
举报内容
In addition to measurement errors, data entry mistakes, natural variability, or unusual events, outliers may indicate missing value codes. Therefore, checking the codebook for your data set should be step one in the process of checking for outliers. For example, 999 is used as a missing value code in many public data sets. I recently encountered 999 and other numeric missing value codes in a public data set published by SAMHSA - the agency within the U.S. Department of Health and Human Services.

已翻译

赞

3 How to handle outliers

When dealing with outliers in regression analysis, there are several options available depending on the nature and purpose of the data and the analysis. Removing outliers may reduce the sample size and introduce bias or loss of information. Alternatively, outliers can be replaced with more reasonable values, such as the mean, median, or mode. Additionally, a mathematical transformation can be applied to the data to reduce the impact of outliers. Finally, a more robust or flexible regression model can be used to account for outliers and reduce their influence. However, each of these options has its own drawbacks and may introduce bias or distortion, change the interpretation of the data, or require more assumptions, parameters, or computations.

添加您的观点

4 How to choose a method

When it comes to handling outliers in regression analysis, there is no definitive rule to follow, as it depends on various factors. Before deciding on a method, you should take the time to understand the data, the outliers, and the method. Explore the data and its characteristics, context, and limitations. Identify the outliers and examine their causes, effects, and patterns. Evaluate the pros and cons of each method and test its performance and validity on your data to ensure that it aligns with your analytical goals and expectations.

添加您的观点

5 How to use tools

There are many tools that can help you detect and handle outliers in regression analysis, such as Excel, R, Python, or Power BI. Depending on the tool and the method, you may need to use different functions, commands, or packages to perform the tasks. For example, in Excel, you can use the SLOPE, INTERCEPT, RSQ, STEYX, or LINEST functions to calculate the regression coefficients and statistics, and use the charts or conditional formatting features to create scatter plots or box plots. In R, you can use the lm, rstandard, cooks.distance, or outliers packages to perform linear regression, calculate standardized residuals and Cook's distance, and remove or replace outliers. In Python, you can use the statsmodels, scipy, or sklearn packages to perform linear regression, calculate standardized residuals and Cook's distance, and apply transformations or robust models. In Power BI, you can use the Power Query Editor or DAX to perform data cleansing, manipulation, or transformation, and use the Analytics pane or R script visuals to create scatter plots or box plots.

添加您的观点

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Business Intelligence

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

Struggling to streamline BI system performance issues?

查看全部

How do you detect and handle outliers in regression analysis?

1

2

3

4

5

6

1 Why outliers matter

2 How to detect outliers

3 How to handle outliers

4 How to choose a method

5 How to use tools

6 Here’s what else to consider

Business Intelligence

给文章评分

感谢您的反馈

更多Business Intelligence相关文章

更多相关阅读内容

How do you detect and handle outliers in regression analysis?

1

2

3

4

5

6

1 Why outliers matter

2 How to detect outliers

3 How to handle outliers

4 How to choose a method

5 How to use tools

6 Here’s what else to consider

Business Intelligence

给文章评分

感谢您的反馈

查看其他技能