Linear Discriminant Analysis

Linear Discriminant Analysis:

1. Linear Discriminant analysis is one of the most popular dimensionality reduction techniques used for supervised classification problems in machine learning. It is also considered a pre-processing step for modeling differences in ML and applications of pattern classification.
2. It is used for modelling differences in groups i.e. separating two or more classes
3. It is used to project the features in higher dimension space into a lower dimension space.?
4. Although the logistic regression algorithm is limited to only two-class, linear Discriminant analysis is applicable for more than two classes of classification problems.

Let Us consider an example

1. For example, we have two classes and we need to separate them efficiently. Classes can have multiple features. Using only a single feature to classify them may result in some overlapping as shown in the below figure. So, we will keep on increasing the number of features for proper classification

Why LDA?

1. Logistic Regression is one of the most popular linear classification models that perform well for binary classification but falls short in the case of multiple classification problems with well-separated classes. While LDA handles these quite efficiently.
2. LDA can also be used in data preprocessing to reduce the number of features just as PCA which reduces the computing cost significantly.
3. LDA is also used in face detection algorithms. In Fisherfaces LDA is used to extract useful data from different faces. Coupled with eigenfaces it produces effective results.

Example

Let's assume we have to classify two different classes having two sets of data points in a 2-dimensional plane as shown in the image:

it is impossible to draw a straight line in a 2-d plane that can separate these data points efficiently but using linear Discriminant analysis; we can dimensionally reduce the 2-D plane into the 1-D plane. Using this technique, we can also maximize the separability between multiple classes.

How Linear Discriminant Analysis (LDA) works?

1. .Linear Discriminant analysis is used as a dimensionality reduction technique in machine learning, using which we can easily transform a 2-D and 3-D graph into a 1-dimensional plane.
2. Let's consider an example where we have two classes in a 2-D plane having an X-Y axis, and we need to classify them efficiently. As we have already seen in the above example that LDA enables us to draw a straight line that can completely separate the two classes of the data points. Here, LDA uses an X-Y axis to create a new axis by separating them using a straight line and projecting data onto a new axis.
3. Hence, we can maximize the separation between these classes and reduce the 2-D plane into 1-D.
4. To create a new axis, Linear Discriminant Analysis uses the following criteria:
5. It maximizes the distance between means of two classes.
6. It minimizes the variance within the individual class.

7.Using the above two conditions, LDA generates a new axis in such a way that it can maximize the distance between the means of the two classes and minimizes the variation within each class.

8.In other words, we can say that the new axis will increase the separation between the data points of the two classes and plot them onto the new axis.

Drawbacks of Linear Discriminant Analysis (LDA)

1. Although, LDA is specifically used to solve supervised classification problems for two or more classes which are not possible using logistic regression in machine learning. But LDA also fails in some cases where the Mean of the distributions is shared. In this case, LDA fails to create a new axis that makes both the classes linearly separable.
2. To overcome such problems, we use?non-linear Discriminant analysis?in machine learning.

Extension to Linear Discriminant Analysis (LDA)

Linear Discriminant analysis is one of the most simple and effective methods to solve classification problems in machine learning. It has so many extensions and variations as follows:

1. Quadratic Discriminant Analysis (QDA):?For multiple input variables, each class deploys its own estimate of variance.
2. Flexible Discriminant Analysis (FDA):?it is used when there are non-linear groups of inputs are used, such as splines.
3. Flexible Discriminant Analysis (FDA):?This uses regularization in the estimate of the variance (actually covariance) and hence moderates the influence of different variables on LDA.

Real-world Applications of LDA

1. Face Recognition
2. Medical
3. Customer Identification
4. For Predictions
5. In Learning