Table of Links Abstract and 1 Introduction 2 Preliminaries 3. Revisiting Normalization 3.1 Revisiting Euclidean Normalization 3.2 Revisiting Existing RBN 4 Riemannian Normalization on Lie Groups 5 LieBN on the Lie Groups of SPD Manifolds and 5.1 Deformed Lie Groups of SPD Manifolds 5.2 LieBN on SPD Manifolds 6 Experiments 6.1 Experimental Results 7 Conclusions, Acknowledgments, and References APPENDIX CONTENTS A Notations B Basic layes in SPDnet and TSMNet C Statistical Results of Scaling in the LieBN D LieBN as a Natural Generalization of Euclidean BN E Domain-specific Momentum LieBN for EEG Classification F Backpropagation of Matrix Functions G Additional Details and Experiments of LieBN on SPD manifolds H Preliminary Experiments on Rotation Matrices I Proofs of the Lemmas and Theories in the Main Paper D LIEBN AS A NATURAL GENERALIZATION OF EUCLIDEAN BN The centering and biasing in Euclidean BN correspond to the group action of R. From a geometric perspective, the standard Euclidean metric is invariant under this group operation. Consequently, it is not surprising that our LieBN algorithm formulated in Alg. 1 serves as a natural generalization of standard Euclidean batch normalization. We formalize this fact in the following proposition. E DOMAIN-SPECIFIC MOMENTUM LIEBN FOR EEG CLASSIFICATION where i is the index of the domain. We follow the official code of SPDDSMBN[5] to implement our DSMLieBN. In a word, the only difference between DSMLieBN and SPDDSMBN is the different way of normalization. Analogous to Thm. 5.3, computations for DSMLieBN under pullback metrics can also be performed by mapping, calculating, and then remapping. [5] https://github.com/rkobler/TSMNet F BACKPROPAGATION OF MATRIX FUNCTIONS Our implementation of LieBN on SPD manifolds involves several matrix functions. Thus, we employ matrix backpropagation (BP) (Ionescu et al., 2015) for gradient computation. These matrix operations can be divided into Cholesky decomposition and the functions based on Eigende-composition. The differentiation of the Cholesky decomposition can be found in Murray (2016, Eq. 8) or Lin (2019, Props. 4). Besides, our homemade BP of the Cholesky decomposition yields a similar gradient to the one generated by autograd of torch.linalg.cholesky. Therefore, during the experiments, we use torch.linalg.cholesky. where ∇XL is the Euclidean gradient of the loss function L w.r.t. X. Matrix K is defined as This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors:
(1) Ziheng Chen, University of Trento;
(2) Yue Song, University of Trento and a Corresponding author;
(3) Yunmei Liu, University of Louisville;
(4) Nicu Sebe, University of Trento. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Preliminaries 2 Preliminaries 3. Revisiting Normalization 3.1 Revisiting Euclidean Normalization 3.1 Revisiting Euclidean Normalization 3.2 Revisiting Existing RBN 3.2 Revisiting Existing RBN 4 Riemannian Normalization on Lie Groups 4 Riemannian Normalization on Lie Groups 5 LieBN on the Lie Groups of SPD Manifolds and 5.1 Deformed Lie Groups of SPD Manifolds 5 LieBN on the Lie Groups of SPD Manifolds and 5.1 Deformed Lie Groups of SPD Manifolds 5.2 LieBN on SPD Manifolds 5.2 LieBN on SPD Manifolds 6 Experiments 6 Experiments 6.1 Experimental Results 6.1 Experimental Results 7 Conclusions, Acknowledgments, and References 7 Conclusions, Acknowledgments, and References APPENDIX CONTENTS APPENDIX CONTENTS A Notations A Notations B Basic layes in SPDnet and TSMNet B Basic layes in SPDnet and TSMNet C Statistical Results of Scaling in the LieBN C Statistical Results of Scaling in the LieBN D LieBN as a Natural Generalization of Euclidean BN D LieBN as a Natural Generalization of Euclidean BN E Domain-specific Momentum LieBN for EEG Classification E Domain-specific Momentum LieBN for EEG Classification F Backpropagation of Matrix Functions F Backpropagation of Matrix Functions G Additional Details and Experiments of LieBN on SPD manifolds G Additional Details and Experiments of LieBN on SPD manifolds H Preliminary Experiments on Rotation Matrices H Preliminary Experiments on Rotation Matrices I Proofs of the Lemmas and Theories in the Main Paper I Proofs of the Lemmas and Theories in the Main Paper D LIEBN AS A NATURAL GENERALIZATION OF EUCLIDEAN BN The centering and biasing in Euclidean BN correspond to the group action of R. From a geometric perspective, the standard Euclidean metric is invariant under this group operation. Consequently, it is not surprising that our LieBN algorithm formulated in Alg. 1 serves as a natural generalization of standard Euclidean batch normalization. We formalize this fact in the following proposition. E DOMAIN-SPECIFIC MOMENTUM LIEBN FOR EEG CLASSIFICATION where i is the index of the domain. We follow the official code of SPDDSMBN[5] to implement our DSMLieBN. In a word, the only difference between DSMLieBN and SPDDSMBN is the different way of normalization. Analogous to Thm. 5.3, computations for DSMLieBN under pullback metrics can also be performed by mapping, calculating, and then remapping. [5] https://github.com/rkobler/TSMNet F BACKPROPAGATION OF MATRIX FUNCTIONS Our implementation of LieBN on SPD manifolds involves several matrix functions. Thus, we employ matrix backpropagation (BP) (Ionescu et al., 2015) for gradient computation. These matrix operations can be divided into Cholesky decomposition and the functions based on Eigende-composition. The differentiation of the Cholesky decomposition can be found in Murray (2016, Eq. 8) or Lin (2019, Props. 4). Besides, our homemade BP of the Cholesky decomposition yields a similar gradient to the one generated by autograd of torch.linalg.cholesky. Therefore, during the experiments, we use torch.linalg.cholesky. where ∇XL is the Euclidean gradient of the loss function L w.r.t. X. Matrix K is defined as This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv Authors: (1) Ziheng Chen, University of Trento; (2) Yue Song, University of Trento and a Corresponding author; (3) Yunmei Liu, University of Louisville; (4) Nicu Sebe, University of Trento. Authors: Authors: (1) Ziheng Chen, University of Trento; (2) Yue Song, University of Trento and a Corresponding author; (3) Yunmei Liu, University of Louisville; (4) Nicu Sebe, University of Trento.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Abstract

Matrix

LieBN as a Natural Generalization of Euclidean BN, And Backpropagation of Matrix Functions

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Brief Review of the Lie Group and the Geometries of SPD Manifolds

5 Best Practices for Integrating with External APIs

Creating Batched Notifications in a Dedicated Time Window for Better UX and Saving Notification Cost

A Lie Group Approach to Riemannian Batch Normalization

A Brief Review of the Lie Group and the Geometries of SPD Manifolds

A Brief Review of the Lie Group and the Geometries of SPD Manifolds

5 Best Practices for Integrating with External APIs

Creating Batched Notifications in a Dedicated Time Window for Better UX and Saving Notification Cost

A Lie Group Approach to Riemannian Batch Normalization

A Brief Review of the Lie Group and the Geometries of SPD Manifolds

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps