What are the most effective methods for Dimensionality Reduction and Feature Selection?

Consider scores of 100 students in exams of 10 subjects given by matrix X of shape (100, 10). Assume that scores are mostly determined by 3 factors – verbal, mathematical and physical ability, given by matrix Y of shape (100, 3). To achieve this, we multiply X with matrix U of shape (10, 3) such that: Y = XU i.e., (100, 10) * (10, 3) --> (100, 3). Every column of U represents a linear combination of exam scores to get “derived” ability score. PCA also acts as feature extraction technique as matrix Y can be used instead of X in fitting a model. However, model explainability may suffer. For PCA to work well, the data needs to lie on linear subspace, else use Non-Linear PCA. We can visualize Word2Vec Embeddings on 2D space using PCA.

Dimensionality reduction is crucial in data science to manage data without losing essential information. Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) are linear methods that maximize data variance and class separation. Factor Analysis (FA) explores underlying factors influencing observed variables. Non-linear techniques like Kernel PCA and t-SNE focus on dealing with complex data structures, optimizing data visualization, and clustering. At the same time, Isometric Mapping (Isomap) emphasizes preserving the global network of the data, finding its use in varied domains like robotics and computer vision.

I used Principal Component Analysis (PCA) in a data project to simplify a complex dataset with numerous customer behavior features. PCA helped reduce the data's dimensionality, making it easier to work with and speeding up our machine learning models. It also uncovered hidden patterns, like the impact of specific product categories on customer behavior, which we might have missed without PCA. This practical application of PCA improved our data analysis and model performance.

Principal component analysis (PCA) is a useful analytical technique and commonly applied in many industries including pharmaceutical sector. However, important to explore the underlying data distribution first. PCA may not capture structure well if data is non-linear.

Consider data (X) and classes (Y), models are of two types: 1. Discriminative: Modelling P(Y | X) 2. Generative: Modelling P(X, Y) i.e., P(Y | X) * P(X), from Bayes' Theorem LDA is generative classifier that not only models decision boundary P(Y | X) between pair of classes but also underlying distribution of data P(X). LDA assumes that training observations are sampled from Gaussian Distribution. Theoretically, we can fit Gaussians for each class using MLE and equate class probabilities to find decision boundaries between each pair of classes. LDA is also a dimension reduction method as it can project data to say 1st two discriminant axes maximizing inter-class and minimizing intra-class separability, reducing the problem to 2D space.

Understanding feature importance is important to be able to create parsimonious and interpretable models. There are many methods of which the main ones are: statistical correlation scores, coefficients calculated as part of linear models, decision trees, permutation importance scores and Neural Networks (by using permutation importance or L1 regularisation techniques). The most common techniques are supervised ones where correlation between dependent and indepent variables are calculated. I have found Decision Trees and Random Forests (Embedded or Intrinsic method) to be particularly helpful, due to the visual nature of the feedback. One should note that different techniques produce different results, domain knowledge is often required.