Beyond Kuramoto Models: Associative Memory and Plastic Synapses in ML Ensembles

Table of Links

Abstract and 1. Introduction

2. Some recent trends in theoretical ML

   2.1 Deep Learning via continuous-time controlled dynamical system

   2.2 Probabilistic modeling and inference in DL

   2.3 Deep Learning in non-Euclidean spaces

   2.4 Physics Informed ML

3. Kuramoto model

   3.1 Kuramoto models from the geometric point of view

   3.2 Hyperbolic geometry of Kuramoto ensembles

   3.3 Kuramoto models with several globally coupled sub-ensembles

4. Kuramoto models on higher-dimensional manifolds

   4.1 Non-Abelian Kuramoto models on Lie groups

   4.2 Kuramoto models on spheres

   4.3 Kuramoto models on spheres with several globally coupled sub-ensembles

   4.4 Kuramoto models as gradient flows

   4.5 Consensus algorithms on other manifolds

5. Directional statistics and swarms on manifolds for probabilistic modeling and inference on Riemannian manifolds

   5.1 Statistical models over circles and tori

   5.2 Statistical models over spheres

   5.3 Statistical models over hyperbolic spaces

   5.4 Statistical models over orthogonal groups, Grassmannians, homogeneous spaces

6. Swarms on manifolds for DL

   6.1 Training swarms on manifolds for supervised ML

   6.2 Swarms on manifolds and directional statistics in RL

   6.3 Swarms on manifolds and directional statistics for unsupervised ML

   6.4 Statistical models for the latent space

   6.5 Kuramoto models for learning (coupled) actions of Lie groups

   6.6 Grassmannian shallow and deep learning

   6.7 Ensembles of coupled oscillators in ML: Beyond Kuramoto models

7. Examples

   7.1 Wahba’s problem

   7.2 Linked robot’s arm (planar rotations)

   7.3 Linked robot’s arm (spatial rotations)

   7.4 Embedding multilayer complex networks (Learning coupled actions of Lorentz groups)

8. Conclusion and References

6.7 Ensembles of coupled oscillators in ML: Beyond Kuramoto models

For the sake of completeness of the exposition, we mention some previous ML experiments with ensembles of coupled phase oscillators. Underline that most of these studies experimented with dynamical systems which do not belong to Kuramoto models.

6.7.1 Ensembles of coupled phase oscillators as associative memories

Several studies proposed to regard ensembles of coupled oscillators as associative memories [134, 135, 136]. This proposal was inspired by the Hopfield model, since ensembles of phase oscillators can be treated as generalizations of the famous Ising model. Ensembles of phase oscillators with symmetric couplings yield gradient flows on tori.

Models proposed as associative memories are not Kuramoto models, since they include couplings through higherorder harmonics. Hence, they are not systems of geometric matrix Riccati ODE’s.

One could think about designing associative memories based on extensions of ensembles of coupled oscillators to higher-dimensional manifolds. This might sound as a promising idea, but the corresponding theory is non-existent at the moment. Indeed, couplings through higher harmonics have been studied only for the ensembles of the classical phase oscillators, but not for generalized oscillators on spheres and other manifolds. Therefore, further investigations in this direction does not seem very promising, due to the total lack of theoretical background.

6.7.2 Swarms with plastic synapses for ML

An intriguing field of study in ML are NN architectures with plastic synapses (adaptive weights). The whole idea is, to a great extent, inspired by Neuroscience. We refer to the recent papers [137, 138] for fairly comprehensive reviews of NN’s with plastic synapses. Networks of this kind can be trained through backpropagation [137].

The central paradigm in networks with plastic synapses is the Hebbian learning rule. This rule, often summarized as "cells that fire together, wire together", imposes that synapses between two neurons get stronger, as their states get closer.

Adaptive synapses obeying Hebbian rules are easily and naturally implemented in networks of Kuramoto oscillators [139]. Furthermore, it is straightforward to extend this model by introducing Kuramoto models on spheres with adaptive couplings [140]. Many models of this kind exhibit potential dynamics.

One can enrich Kuramoto models on orthogonal groups and spheres by assuming that the couplings are adaptive and setting up the learning rule. In order to increase the representative power one can combine Hebbian and anti-Hebbian learning rules. In such a way one can construct the models representing coupled actions of the group SO(d) or complicated probability distributions on spheres.

Notice that systems with adaptive couplings are not suitable for encoding hyperbolic isometries. Hence, actions of the Lorentz groups can not be modeled in this way.