paint-brush
The Math Behind Selective State Space Modelsby@serialization

The Math Behind Selective State Space Models

by The Serialization Publication
The Serialization Publication HackerNoon profile picture

The Serialization Publication

@serialization

We cover the most cutting edge academic research and expert...

December 18th, 2024
Read on Terminal Reader
Read this story in a terminal
Print this story
Read this story w/o Javascript
Read this story w/o Javascript
tldt arrow

Too Long; Didn't Read

This section examines the mechanics of Selective SSMs, detailing the discretization process, the role of learnable biases, and how the zero-order hold (ZOH) formulas shape efficient AI recurrences.
featured image - The Math Behind Selective State Space Models
1x
Read by Dr. One voice-avatar

Listen to this story

The Serialization Publication HackerNoon profile picture
The Serialization Publication

The Serialization Publication

@serialization

We cover the most cutting edge academic research and expert blog posts on serialization. Also big fans of the Serial pod

Learn More
LEARN MORE ABOUT @SERIALIZATION'S
EXPERTISE AND PLACE ON THE INTERNET.

Authors:

(1) Albert Gu, Machine Learning Department, Carnegie Mellon University and with equal contribution;

(2) Tri Dao, Department of Computer Science, Princeton University and with equal contribution.

Abstract and 1 Introduction

2 State Space Models

3 Selective State Space Models and 3.1 Motivation: Selection as a Means of Compression

3.2 Improving SSMs with Selection

3.3 Efficient Implementation of Selective SSMs

3.4 A Simplified SSM Architecture

3.5 Properties of Selection Mechanisms

3.6 Additional Model Details

4 Empirical Evaluation and 4.1 Synthetic Tasks

4.2 Language Modeling

4.3 DNA Modeling

4.4 Audio Modeling and Generation

4.5 Speed and Memory Benchmarks

4.6 Model Ablations

5 Discussion

6 Conclusion and References


A Discussion: Selection Mechanism

B Related Work

C Mechanics of Selective SSMs

D Hardware-aware Algorithm For Selective SSMs

E Experimental Details and Additional Results

C Mechanics of Selective SSMs

image


The discretization step size is


image


where we observe that the parameter can be viewed as a learnable bias and folded into the linear projection. Now applying the zero-order hold (ZOH) discretization formulas:


image


Thus the final discrete recurrence (2a) is


image


as desired.


This paper is available on arxiv under CC BY 4.0 DEED license.


L O A D I N G
. . . comments & more!

About Author

The Serialization Publication HackerNoon profile picture
The Serialization Publication@serialization
We cover the most cutting edge academic research and expert blog posts on serialization. Also big fans of the Serial pod

TOPICS

THIS ARTICLE WAS FEATURED IN...

Permanent on Arweave
Read on Terminal Reader
Read this story in a terminal
 Terminal
Read this story w/o Javascript
Read this story w/o Javascript
 Lite
Also published here
Hackernoon
X
Threads
Bsky