Logistic Regression on Riemann Manifolds

Traditional linear models in machine learning, such as logistic regression, struggle to grasp the complex characteristics of data in very high dimensions.?

Symmetric Positive Definite manifolds improve the output quality of logistic regression by enhancing feature representation in a lower-dimensional space.

Introduction
Logistic regression on manifolds
        Logistic regression
        SPD manifolds
        Logarithmic map
        Affine invariant Riemannian metric
        Log Euclidean Riemannian metric
Implementation
        Setup
        Data generation
        Manifold creation
Validation
        Euclidean space
        Classification on SPD manifolds
References
Appendix        

What you will learn:?How to implement and validate a binary logistic regression classifier on SPD manifolds using affine invariant and log Euclidean metrics.

The complete article, featuring design principles, detailed implementation, in-depth analysis, and exercises, is available in Insights into Logistic Regression on Riemannian Manifolds

Notes:?

• Environments:?Python ?3.10.10, Geomstats 2.7.0, Scikit-learn 1.4.2
• The implementation relies on Geomstats, an open-source, object-oriented library following Scikit-Learn’s API conventions to gain hands-on experience with Geometric Learning. It is described in article Introduction to Geomstats for Geometric Learning
• This article assumes that the reader is somewhat familiar with differential and tensor calculus [ref?1]. Please refer to our previous articles related to geometric learning [ref?2,?3,?4].
• Source code is available at??Github.com/patnicolas/Data_Exploration/manifolds
• To enhance the readability of the algorithm implementations, we have omitted non-essential code elements like error checking, comments, exceptions, validation of class and method arguments, scoping qualifiers, and import statements.

Introduction

This article is the eight installments of our ongoing series focused on geometric learning. In this installment, we utilize the Geomstats Python library [ref.?5].

Note:?Summaries of my earlier articles on this topic can be found in the Appendix.

The primary goal of learning?Riemannian geometry?is to understand and analyze the properties of curved spaces that cannot be described adequately using Euclidean geometry alone.?

Using logistic regression for classification on low-dimensional data manifolds offers several benefits:

Simplicity and interpretability: The model provides clear insights into the relationship between the input features and the probability of belonging to a certain class.

Efficiency: On low-dimensional manifolds, logistic regression is computationally efficient.

Good performance in linearly separable cases: The logistic regression performs exceptionally well if the data in the low-dimensional manifold is linearly separable.

Robustness to overfitting: In lower-dimensional spaces, the risk of a simpler model such as the logistic regression to overfit is generally reduced.

Support for non-linear boundaries: Although linear, the logistic regression can handle non-linear boundaries in low-dimensional space than Euclidean space.

This article relies on the?Symmetric Positive Definite?(SPD) Lie group of matrices as our manifold for evaluation. We will introduce, review or describe:

1. Logistic regression as a binary classifier
2. SPD ?matrices
3. Logarithms and exponential maps on manifolds introduced in Differentiable Manifolds
4. Riemannian metrics associated to SPD
5. Implementation of binary logistic regression using Scikit-learn and Geomstats Python libraries
6. Verification using randomly generated SPDs and cross-validation.

Logistic regression on manifolds

Logistic regression

Let's review the ubiquitous binary logistic regression. For a set of two classes?C = {0, 1}?the probability of predicting the correct class given a features set?x?and a model weights?w?is defined by sigmoid,?sigm, transform:

The binary classifier is then defined as?

C := 1 <=> p(C=1|x, w) >= 0.5and?C := 0 <=> p(C=1|x, w) < 0.5.

For an introduction to basic logistic regression and its implementation for beginners, check out this detailed guide: Logistic Regression Explained and Implemented in Python?(ref?6).

SPD manifolds

Let's introduce our manifold defined as the group of symmetric positive definite (SPD) matrices.?

SPD matrices are used in a wide range of applications:

• Diffusion tensor imaging (analysis of diffusion of molecules and proteins)
• Brain connectivity
• Dimension reduction through kernels
• Robotics and dynamic?systems
• Multivariate principal component analysis
• Spectral analysis and signal reconstruction
• Partial differential equations numerical methods
• Financial risk management.

A square matrix?A?is symmetric if it is identical to its transpose, meaning that if aij are the entries of?A, then?aij=aji. This?implies that?A?can be fully described by its?upper triangular elements, aij

A square matrix?A?is positive definite if, for every non-zero vector?b, the product?bTAb?>= 0.

If a matrix?A?is both symmetric and positive definite, it is referred to as a symmetric positive definite (SPD) matrix. This type of matrix is extremely useful and appears in various real-world applications. A prominent example in statistics is the covariance matrix, where each entry represents the covariance between two variables (with diagonal entries indicating the variances of individual variables). Covariance matrices are always positive semi-definite (meaning?bTAb≥0), and they are positive definite if the covariance matrix has full rank, which occurs when each row is linearly independent from the others.The collection of all SPD matrices of size?n×n?forms a manifold.

Logarithmic map?

As discussed in Differentiable Manifolds the exponential & logarithmic maps and parallel transportation are crucial for Riemannian approaches in machine learning. On any manifold, distance, dist(x, y)?are defined as geodesics that correspond to straight lines in Euclidean space.

Let consider a vector x to y on a tangent space at point y. Operations on the point on the manifold ?rely on the exponential map (projection) onto the manifold.?

The table below show the equivalent operations between Euclidean space and manifolds.

Operations Euclidean Manifold

In the case of the binary logistic regression, the prediction on the manifold is defined by the exponential map?expx:

Let select two Riemannian metrics for the SPD manifolds [ref?7].

Affine invariant?Riemannian?metric

For any two symmetric positive definite (SPD) matrices ?? and ??, the Affine Invariant Riemannian Metric (AIRM) between them is defined as:?

Log-euclidean?Riemannian?metric

Given a symmetric positive definite matrix SPD at point S and a tangent space TsSPD, the logarithmic and exponential maps can be expressed as:?

Implementation

Further applications and evaluations are available in Insights into Logistic Regression on Riemannian Manifolds

Setup

First, let's create a data class, SPDTestData that encapsulates the training features X and label y. This class will be used to validate our implementation of logistic regression on SPD manifolds using various metrics, as well as in Euclidean space.

The?flatten?method vectorizes each 2-dimension SPD matrix entry in the training set to be processed by the?scikit-learn?cross validation function.

We wrap the generation of random data, SPD manifolds and the evaluation of various metrics in the class?BinaryLRManifold.

The generation of the labeled training set uses the?numpyrandom values generation method.

Data generation

The method?__generate_spd_data?create symmetric positive definite n_features x n_features matrices by computing eigenvalues for the diagonal component, reducing the upper triangle values and replicating them to the lower triangle.

Manifold generation

Creating an SPD matrix is straightforward:

1. Instantiate the Geomstats SPDMatrices class
2. Equip it with a Riemannian metric.

The following code snippet creates two SPD manifolds with affine metric and log Euclidean Riemannian metrics.

Validation

The initial phase involves verifying our implementation of the metrics related to SPD manifolds. This is achieved by calculating the cross-validation score for SPD matrices containing random values between [0, 1] and ensuring that the average score approximates 0.5.

Euclidean space

We utilize the logistic regression class and the cross_validate method from Scikit-learn once the contents of the matrix have been converted into a vector form.

The test code used a training set of 6000 samples of 16 x 16 (256) SPD matrices. The binary logistic regression in the Euclidean space as a mean cross validation score of?0.487?instead of?0.5.

Output:

Cross validation: [0.478 0.513 0.497 0.474 0.471] with mean: 0.487        

Classification on SPD manifold

To utilize scikit-learn's cross-validation features, the SPD matrix must first be differentiated on its tangent space before applying logistic regression. These two steps are executed using a scikit-learn pipeline.

We employ the same training setup as used in the evaluation of logistic regression in Euclidean space, but we apply the log Euclidean (SPDLogEuclideanMetric) and affine invariant (SPDAffineInvariant) metrics. The mean values of the cross-validation scores are 0.492 and 0.5, respectively, which significantly improve upon the results from the Euclidean scenario.

Output for Log Euclidean metric:

Cross validation: [0.495  0.504 0.498 0.491 0.470] with mean: 0.492        

Output for affine invariant metric

Cross validation: [0.514  0.490 0.490 0.490 0.504] with mean: 0.500        

Thank you for reading this article! I hope you found this overview insightful. For a detailed exploration of the topic, check out?Insights into Logistic Regression on Riemannian Manifolds

References

[1]?Tensor Calculus - Eigenchris

[2]?Geometric Learning in Python: Basics

[3]?Differentiable Manifolds

[4]?Vector and Covector Fields

[5]?Introduction to Geometric Learning in Python with Geomstats

[6]?Logistic Regression Explained and Implemented in Python.

[7]?Log-Euclidean Metric Learning on Symmetric Positive Definite Manifold with Application to Image Set Classification

[8] ?Insights into Logistic Regression on Riemannian Manifolds

Patrick Nicolas has over 25 years of experience in software and data engineering, architecture design and end-to-end deployment and support with extensive knowledge in machine learning.? He has been director of data engineering at Aideo Technologies since 2017 and he is the?author of "Scala for Machine Learning", Packt Publishing ISBN 978-1-78712-238-3 and Hands-on Geometric Deep Learning newsletter

Appendix

List of articles related to Geometric Learning on this newsletter:

• Geometric Learning in Python: Basics introduces differential geometry as an applied to machine learning and its basic components.
• Differentiable Manifolds describes manifold?components such as?tangent vectors, geodesics with implementation in Python for Hypersphere using the?Geomstats?library.
• Intrinsic Representation Reviews the various coordinates system using extrinsic and intrinsic representation.
• Vector and Covector Fields describes vector and co-vector fields with Python implementation in 2 and 3-dimension spaces.
• Vector Operators in Python illustrates the differential operators, gradient, divergence, curl and laplacian using?SymPy?library.
• Non-linear Functional Data Analysis describes the key elements of non-linear functional data analysis to analysis?curves, images, or functions in very ?high-dimensional spaces
• Riemann Metric & Connection for Geometric Learning reviews the key components of the Levi-Civita connection and its implementation in Python using Geomstats library.
• Riemann Curvature for Geometric Learning in Python Introduces the concept of curvature tensor in the context of the Levi-Civita connection.
• K-Means on Riemann Manifolds ?compares the implementation of k-means algorithm on Euclidean space using?#Scikit-learn and hypersphere using?Geomstats.

#Geomstats #ScikitLearn #RiemannianManifold #GeometricLearning #LogisticRegression