Reinforcement Learning on Non-Euclidean Spaces: Swarms, Spheres, and Hyperbolic RL

Table of Links

Abstract and 1. Introduction

2. Some recent trends in theoretical ML

   2.1 Deep Learning via continuous-time controlled dynamical system

   2.2 Probabilistic modeling and inference in DL

   2.3 Deep Learning in non-Euclidean spaces

   2.4 Physics Informed ML

3. Kuramoto model

   3.1 Kuramoto models from the geometric point of view

   3.2 Hyperbolic geometry of Kuramoto ensembles

   3.3 Kuramoto models with several globally coupled sub-ensembles

4. Kuramoto models on higher-dimensional manifolds

   4.1 Non-Abelian Kuramoto models on Lie groups

   4.2 Kuramoto models on spheres

   4.3 Kuramoto models on spheres with several globally coupled sub-ensembles

   4.4 Kuramoto models as gradient flows

   4.5 Consensus algorithms on other manifolds

5. Directional statistics and swarms on manifolds for probabilistic modeling and inference on Riemannian manifolds

   5.1 Statistical models over circles and tori

   5.2 Statistical models over spheres

   5.3 Statistical models over hyperbolic spaces

   5.4 Statistical models over orthogonal groups, Grassmannians, homogeneous spaces

6. Swarms on manifolds for DL

   6.1 Training swarms on manifolds for supervised ML

   6.2 Swarms on manifolds and directional statistics in RL

   6.3 Swarms on manifolds and directional statistics for unsupervised ML

   6.4 Statistical models for the latent space

   6.5 Kuramoto models for learning (coupled) actions of Lie groups

   6.6 Grassmannian shallow and deep learning

   6.7 Ensembles of coupled oscillators in ML: Beyond Kuramoto models

7. Examples

   7.1 Wahba’s problem

   7.2 Linked robot’s arm (planar rotations)

   7.3 Linked robot’s arm (spatial rotations)

   7.4 Embedding multilayer complex networks (Learning coupled actions of Lorentz groups)

8. Conclusion and References

6.2 Swarms on manifolds and directional statistics in RL

References on RL on non-Euclidean spaces are very sparse. Researchers only recently started to apply RL techniques to the learning over data sets with non-Euclidean geometries. Reward-based update of deterministic and stochastic policies on Riemannian manifolds has potential applications in different fields. Hence, Markov decision processes with non-Euclidean space of states (and actions) present an important challenge for ML. However, rather than developing the general theory, it seems more advantageous to start with some paradigmatic examples in order to elaborate conceptual approaches. Two recent studies [59, 64] considered RL algorithms on spheres and special orthogonal groups. Both studies proposed the Bingham distribution for implementation of stochastic policies.

Parametrization of stochastic policies by spherical Cauchy and von Mises-Fisher distributions in RL is still to be explored. In our point of view, the idea of using spherical Cauchy distributions in RL has a great potential, given all nice properties of this family. We will discuss this idea on some illustrative examples in the next Section.

We also mention a very recent paper [118] on hyperbolic RL. This paper proposes deep RL algorithms for designing more efficient policies that model latent representations in the hyperbolic space. Like most of ML algorithms in hyperbolic geometries this method exploits the framework of gyrovector spaces for operations in hyperbolic spaces.