From Eigendecomposition to Determinant: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 3/3

For understanding the mathematics for machine learning algorithms, especially deep learning algorithms, it is essential to build up the mathematical concepts from foundational to more advanced. Unfortunately, Mathematical theories are too hard/abstract/dry to digest in many cases. Imagine you are eating a pizza, it is always easier and more fun to go with a coke.

The purpose of this article is to provide intuitive examples for fundamental mathematical theories to make the learning experience more enjoyable and memorable, which is to serve chicken wings with beer, fries with ketchup, and rib-eye with wine.

The 3-course fundamental mathematics for machine learning meal is organized as follows:

From Scalar to Tensor: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 1/3

• What are Scalar, Vector, Matrix and Tensor?
• Addition between Scalar, Vector, and Matrix
• Multiplication between Scalar, Vector, and Matrix
• Identity and inverse matrix
• Diagonal matrix and symmetric matrix

From Norm to Orthogonality: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 2/3

• 1-Norm, 2-Norm, Max Norm of Vectors
• Orthogonal and orthonormal vectors
• Orthogonal matrix

From Eigendecomposition to Determinant: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 3/3

• Eigendecomposition of matrix: eigenvalue and eigenvector
• The trace operator
• Determinant of square matrix

In this article, we will go through the part 3/3, From Eigendecomposition to Determinant with intuitive examples.

Eigendecomposition of matrix: eigenvalue and eigenvector

Why we need decomposition? If we want to discover the nature of something, decomposition is an efficient and practical approach. Assuming you just made a new friend, decomposing him/her into the following four components can help you quickly establish a deeper understanding of the specific person.

The same methodology in math. An integer can be decomposed into prime factors, for example, 20 = 2 * 2 * 5, which means 20 is not divisible by 3, and any integer multiple of 20 will be divisible by 5.

Eigendecomposition is a way of decomposing a matrix into a set of eigenvectors and eigenvalues. A nonzero vector v is an eigenvector of a square matrix A if it satisfies the eigenvalue equation:

where λ is a scalar known as the eigenvalue corresponding to vector v.

Eigendecomposition of a matrix A is given by

where V is a square matrix whose i-th column is the i-th eigenvector of matrix A, and diag(λ) is the diagonal matrix whose diagonal elements are corresponding eigenvalues. For example, a real matrix:

may be decomposed into a diagonal matrix through multiplication of a non-singular matrix V:

The above vector equations can be represented by a single vector equation:

By shifting λv to the left hand side, we can have

where vector v is non-zero because matrix V is non-singular. Therefore,

According to the determinant definition of matrix, we can have:

Putting the solutions back into the above vector equations:

Solving the equations, we have

Thus the matrix V required for the eigendecomposition of A is:

For example,

The trace operator

In linear algebra, the trace of a square matrix A is defined to be the sum of the elements on the main diagonal:

For example, there is a 3*3 matrix A,

The trace of matrix A is

Some basic properties of trace operator as follows:

More generally, the trace is invariant to the moving to moving the last factor into the first position, which is termed as cyclic permutation, i.e.,

Determinant of square matrix

What is determinant? Why we need to understand how to calculate the determinant of a matrix? What is the meaning of the value of determinant in machine learning?

The determinant of a square matrix, denoted det(A), is a value that can be computed from the elements of the matrix. For a 2*2 matrix, its determinant is:

For a 3*3 matrix, the determinant is defined as

In the case of a 4*4 matrix, the determinant is

(Above photo credit to Wikipedia)

The determinant is equal to the product of all the eigenvalues of a matrix that we can use determinant to map a matrix to a real scalar. You can use numpy.linalg.det to compute the determinant of an array.

(Above photo credit to Scipy)

Determinant is an important mathematical concept in machine learning algorithms, for example, the widely used dimensionality reduction solution: principal components analysis (PCA). As we known, a matrix can be seen as a linear transformation of the space, the value of its determinant can be thought as the multiplicative change you get when transform the space with this matrix, which could be a rotation, a rescaling, or a change in orientation, etc.

Congratulations! You have completed the 3rd part and also the last piece of Fundamental Mathematics for Machine Learning with Intuitive Examples.

“Wherever there is number, there is beauty.” — Proclus

The beauty of fundamental mathematical theories is that it will never ever be out of date. Ten years later, in 2028, software developers and data scientist might not use Python or TensorFlow any more. But scalar is still scalar, matrix is still matrix, and tensor is still tensor. The necessary and sufficient conditions for the inverse of a matrix would be the same as they are today. Fundamental mathematics can survive the passing of time. You might invest one week or one month on this series. Fortunately, the gain is the knowledge and skill that you could apply for the entire life because — “Mathematics is a way of thinking.”