登录查看更多内容

How do you avoid overfitting and underfitting your predictive model due to feature engineering and selection?

由人工智能和领英社区提供技术支持

Feature engineering and selection are crucial steps in building a predictive model that can generalize well to new data. However, they also pose the risk of overfitting or underfitting your model, which can lead to poor performance and unreliable predictions. In this article, you will learn what overfitting and underfitting are, how to detect them, and how to avoid them by applying some best practices in feature engineering and selection.

此文章中的业界达人

由社区从 5 条内容中精选。了解更多

1 What is overfitting and underfitting?

Overfitting occurs when your model learns too much from the training data and fails to capture the underlying patterns of the problem. This makes your model too complex and sensitive to noise, outliers, or irrelevant features. As a result, your model performs well on the training data, but poorly on the validation or test data, which indicates a high variance and low bias. Underfitting, on the other hand, occurs when your model learns too little from the training data and fails to capture the essential features of the problem. This makes your model too simple and rigid, and unable to adapt to new or complex data. As a result, your model performs poorly on both the training and the validation or test data, which indicates a low variance and high bias.

添加您的观点

Eugenio Zuccarelli

AI Leader for Fortune 100 | Forbes Under 30 | Fortune Under 40 | Author of “Intelligenza Artificiale” (Mondadori) | TEDx Speaker | LinkedIn Top Voice | MIT, Harvard, Imperial College
举报内容
Having way more features than samples (what's usually called "The Curse of Dimensionality") is one of the key reasons overfitting happens. The model tends to find a combination of features that uniquely identifies samples, leading to low bias and high variance (the model has high performance but generalizes poorly). Choosing simpler models and removing features through correlation analyses and regularization usually does the trick.

已翻译

赞
Joy J.

Medfuse Analytics Leader, Founder @ Syneractiv | Marketing Analytics, Data Strategy Advisor
举报内容
Overfitting can be caused by different reasons, but a frequent cause is data sparsity, i.e. when some or all features have only a few non-zero values (typically less than 1/3rd). Such overfitted models built with sparse features do not generalize well beyond the liited range of observed data. Overfitting results in low training data error rates. Underfitting can also be caused by different reasons, but a common one is under-specification- the model doesn't have enough sophisticationn or complexity to represent the observed data. Underfitting results in high training data error.

已翻译

赞

2 How to detect overfitting and underfitting?

One of the most common ways to detect overfitting and underfitting is to use a train-test split or a cross-validation technique to evaluate your model on different subsets of data. By comparing the training and validation or test scores, you can identify if your model is overfitting or underfitting. For example, if your model has a high training score but a low validation or test score, it is likely overfitting. If your model has a low training score and a low validation or test score, it is likely underfitting. If your model has a high training score and a high validation or test score, it is likely well-fitted.

添加您的观点

Joel Atkins

Data Science Leader | Actuary (FCAS) | Drawing Actionable Insights from Data
举报内容
I think cross-validation and train-test are both less than ideal. In cross validation, if you have n folds, than any two folds have (n-2)/(n-1) data in common and only 1/(n-1) data is distinct. Train-test splits the data once and anything that works for those two splits gets in the model. I prefer bootstrap, so I can see how stable my model is across 100 or so samples.

已翻译

赞

3 How to avoid overfitting?

There are several ways to avoid overfitting your predictive model due to feature engineering and selection. You can reduce the number of features by using correlation analysis, feature importance, or dimensionality reduction. Additionally, you can regularize the model with techniques such as Lasso, Ridge, or Elastic Net. Moreover, you can add more data by utilizing data augmentation, synthetic data generation, or external data sources. Finally, you can use ensemble methods such as bagging, boosting, or stacking to combine multiple models and reduce the variance. All of these techniques may help improve the generalization and accuracy of your model.

添加您的观点

Art C.

Chief Data Science and Analytics Officer @ The Hendry Partnership | Marketing, Digital Media, & Sales Strategy and Implementation expert leveraging AI/ML models for Media Planning, Optimization & Forecasting.
举报内容
To avoid over fitting, reduce the number of variables (features), and to use a hold-out sample to test accuracy of the model versus looking at historical fits.

已翻译

赞

4 How to avoid underfitting?

To avoid underfitting your predictive model due to feature engineering and selection, you can increase the number of features by using techniques such as feature extraction, feature construction, or feature interaction. Additionally, you can increase the model complexity with polynomial features, kernel methods, or deep learning. Lastly, you can tune the model parameters with grid search, random search, or Bayesian optimization to find the optimal values for the hyperparameters of your model, which may affect the trade-off between bias and variance and improve the performance of your model.

添加您的观点

5 How to balance feature engineering and selection?

Feature engineering and selection are not one-time tasks, but an iterative process that requires experimentation and evaluation. To achieve the best results, define a clear objective and metric for your predictive model and use it to guide your feature engineering and selection decisions. Additionally, use a systematic and consistent approach, such as a pipeline or a framework, to compare different options and track their results. Validation or test sets or cross-validation techniques should be used to assess the impact of feature engineering and selection on your model performance - avoiding using the same data for both training and testing. You should also combine domain knowledge, data exploration, and statistical analysis to understand your data and features in order to generate hypotheses and insights that can inform your feature engineering and selection choices. Finally, use intuition, creativity, experimentation, feedback, and learning to generate and test new features while refining them as you go.

添加您的观点

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Jack Dyer

Tech Leader? Under pressure to drive revenue with data and AI? Get my 3-part strategy. Dog Dad to Benny the spaniel. Massive cricket fan. Founder @ Leftshifted
举报内容
We can use out of sample testing to help build confidence that our trained models will be effective with brand new data, that the models hadn’t had the chance to learn from at all. We should also monitor performance in production, after deploying models live. We are looking for consistency between outputs in training environments and production results, to show that we trained models in a way that mirrored reality, and avoided data leak.

已翻译

赞

Predictive Modeling

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How do you avoid overfitting and underfitting your predictive model due to feature engineering and selection?

1

2

3

4

5

6

1 What is overfitting and underfitting?

2 How to detect overfitting and underfitting?

3 How to avoid overfitting?

4 How to avoid underfitting?

5 How to balance feature engineering and selection?

6 Here’s what else to consider

Predictive Modeling

给文章评分

感谢您的反馈

更多Predictive Modeling相关文章

更多相关阅读内容