Feature selection is a crucial step in machine learning, as it can improve the performance, interpretability, and efficiency of your models. But how do you choose the best features from a large and complex dataset? In this article, you will learn about the different types of feature selection methods and how you can use them.
Filter methods are based on the statistical properties of the features, such as correlation, variance, or mutual information. They do not involve any learning algorithm, and they rank the features according to a certain criterion. Filter methods are fast, scalable, and independent of the model, but they may ignore the interactions and dependencies among the features. You can use filter methods as a preprocessing step to reduce the dimensionality of your data and select the most relevant features.
Feature Engineering and feature selection are most important part of building a model, as we know garbage in is garbage out. So while doing feature selection you can use multiple methodologies few I am list below: - Variance threshold (take a cut-off of let say 20%) indicating if variables lacks variance than its a ~constant variable and a constant variable is of no use for training - General EDA of feature impact on target variable in training set - check for data leakage - multicollinearity and VIF, etc.
Filter methods are a category of feature selection techniques used to select relevant features from a dataset based on their intrinsic characteristics, without involving a ML algos. - scoring techniques like mutual information, chi-squared test, correlation coefficient, and variance analysis can help to assign a score to each feature in from dataset. - post features are scored, each of them are ranked from most relevant to least relevant, features with higher scores are considered more important for the task. In turn, it helps which features to keep/discard. - Using at threshold value, helps to determine how many of the top-ranked features to select. In turn, helps in controlling the dimensionality of the data.
In addition, another simple filter method is: "drop constant and maybe also [almost] constant columns. " (The latter one can get tricky if the classes are not balanced and that almost-constant column is highly associated with the target value)
I approach these filters as follows: 1. Columnar Filters: > One of the easiest methods is to identify zero variance (i.e. the entire column has the exact same value) and near-zero variance columns (i.e. there's very little information to be gained from the column > Linear combinations: Run correlation plots to identify columns with very high correlations (>0.9?) (Alternatively, identify near-zero eigenvalues on the correlation matrix). Determine if you can drop some columns or run dimension reduction using PCA. 2. Row filters: Identify rows with high % of missing values. While you may be able to apply imputation methods, if you have rows with 80%+ missing values in all the columns, perhaps you'd better filter them out.
Wrapper methods use a learning algorithm to evaluate the features and select the optimal subset that maximizes the model performance. They can capture the interactions and dependencies among the features, and they are tailored to a specific model and objective function. However, wrapper methods are computationally expensive, prone to overfitting, and may not generalize well to other models. You can use wrapper methods when you have a small or medium-sized dataset and a clear evaluation metric.
I would say wrapper methods for feature selection are a more sophisticated approach compared to filter methods. - different subsets of features are evaluated using ML algo. Here, each subsets can range from a single feature to the entire feature set. - performance metrics like accuracy, F1-score, or cross-validation error, can be used to assess the quality of the model on each subset of features. - Search strategies like forward selection, backward elimination, and recursive feature elimination (RFE) iteratively add or remove features based on their impact on model performance. Wrapper method typically employ one of these to explore feature space.
I'll restrict my commentary here to the feature selection approach popular in most regression literature, viz. 'best subsets selection' : forward/backward/hybrid selection approaches using some metric like Mallows Cp, AIC, BIC or adjusted Rsq. Outside toy/academic examples, I haven't seen these methods produce quality models. Furthermore, a white paper published by SAS proved that these methods fall woefully short, resulting in misspecified models in 60%+ cases even on small controlled datasets.
Embedded methods combine the advantages of filter and wrapper methods, as they perform feature selection during the training process of the model. They use regularization techniques, such as Lasso or Ridge, to penalize the complexity of the model and shrink the coefficients of the less important features. Embedded methods are efficient, robust, and adaptive to the model, but they may be less interpretable and more sensitive to the choice of hyperparameters. You can use embedded methods when you have a large or high-dimensional dataset and a suitable regularization method.
The part "such as Lasso or Ridge, to penalize the complexity of the model " does not sound correct to me because those parts make the model more complex compared to a linear regression. It allows user to apply more complex model to data while avoiding the overfitting issue.
Hybrid methods are combinations of filter, wrapper, and embedded methods that aim to achieve better results than any single method. They can use different strategies, such as sequential, parallel, or iterative, to integrate the feature selection methods and the learning algorithm. Hybrid methods can balance the trade-offs between speed, accuracy, and complexity, and they can handle different types of data and models. You can use hybrid methods when you want to explore different options and optimize your feature selection process.
Hybrid methods in feature selection bring a powerful blend of filter, wrapper, and embedded techniques, surpassing the capabilities of any individual method. They employ strategies like sequential, parallel, or iterative approaches to seamlessly integrate feature selection methods with learning algorithms. This integration achieves a delicate balance between speed, accuracy, and complexity, making hybrid methods adaptable to various data types and model complexities.
Feature importance is a measure of how much a feature contributes to the prediction of the model. It can be calculated using different methods, such as permutation, Shapley values, or feature weights. Feature importance can help you understand the logic and behavior of your model, as well as identify the most influential features. You can use feature importance to validate, explain, and improve your feature selection results.
One of the most universal ways to measure feature importance is by removing the features from the model one by one and evaluating how model performance changed after that.
One of my favorite approaches to quickly identify important features on small/medium sized models is to plot the beta coefficients of a regression model when the input data is scaled (normalized). A dot plot of these (sorted)beta coefficients, tells a fantastic story of not only which features and interactions are most important, but also the direction (sign) of their contribution.
Feature engineering is the process of creating new features from existing ones, or transforming the features to make them more suitable for the model. It can enhance the quality and diversity of your data, as well as increase the predictive power and interpretability of your model. You can use feature engineering to extract more information, reduce noise, or deal with missing values or outliers in your data.
There are so many nice tools and algorithms out there to assist you in feature selection. However, I find that engineering your own clever and relevant features is the best way to increase model performance. They way to do is to talk, talk, talk to the business or organisation you are working for. Make sure you understand their goals, processes, operations and challenges. You will need to become a minor-domain expert, really dive into the subject matter. You will find that together with your stakeholders you will come up with new features organically. And these features are usually very relevant to model performance! Because in the end, real world dynamics will mirror the data and vice versa. This will "help" an ML model make sense of it.
MLOps Tech Lead | 10+ years in Data & AI | Databricks Beacon | Public speaker
What is often underestimated in feature engineering (and I consider it feature engineering too) is the selection of the relevant data to train the model on. For example, if you want to build a cross-sell model for a supermarket, you do not want to use all available data you have on customer behaviour over the years. It may hurt your model performance. You need to find the sweet spot of the time period to train the model on (for example, only using the data on the last 2 months might better than using the data on last 2 years)
Feature engineering, imho, is simultaneously the hardest part of building a solid performing machine learning model, while also being often the most overlooked in most ML training materials. It's often I find that top performing predictive models on Kaggle aren't using any fancy overtly complex ML pipeline (it's often an ensemble of xgboost, catboost and the like). However, what separates the leaders on the board is the quality of feature engineering. Feature engineering becomes the hardest nut to crack because it requires you to become somewhat of a domain expert, which in turn means you have to poke & prod not only the data, but also the stakeholders & domain experts in the company. Painstaking, slow process? Yes. Value added? 100%.