While everyone is focusing on the hot news about Microsoft's acqui-hiring Inflection AI in disguise and the shakeup at Stability AI, we'd like to concentrate on the exciting developments unfolding in the world of model architectures. For the hot news, check the Usual Suspects © section below. – a new architecture that rivals the famous Transformer-based models. Mamba's innovations address significant challenges in processing long sequences, a problem that has limited traditional models. Now, let's talk about Mamba Mamba leverages state-space models (SSMs)*, particularly excelling with its incorporation of Structured State Space (S4) models into a large language model (LLM) framework. This integration allows Mamba to achieve linear complexity scaling with sequence length, marking a significant advancement over the quadratic scaling seen in traditional Transformer-based models. So what is it? Its streamlined architecture incorporates selective SSM layers, enhancing both efficiency and flexibility. As a result, Additionally, Mamba efficiently processes extremely long sequences, surpassing earlier models in performance. it benefits from hardware-aware optimizations, maximizing the potential of contemporary GPU architectures. This means you can process much longer sequences without hitting memory or compute bottlenecks. Think about applications like genomic analysis, long-form content generation, and complex multi-modal data processing, all becoming more feasible with Mamba's power. *State-space models are mathematical frameworks that describe a system's dynamics in terms of its state variables and observations, capturing the evolution and uncertainty of processes over time.  SSMs are known for efficiency with long sequences. Mamba's ability to efficiently process long sequences while maintaining competitive performance has fueled research interest in adapting and extending the architecture for various domains. It seems Mamba’s architecture is getting more attention (Attention is all you need ;) – last week, three papers showcased exciting developments. The paper, , makes Mamba more suitable for deployment on resource-constrained devices by introducing an efficient 2D scanning method and a dual-pathway module for balanced global-local feature extraction. Results show a significant reduction in FLOPs while maintaining strong accuracy. EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba The paper, , extends Mamba to be a multi-modal large language model capable of jointly reasoning over vision and language. Experiments demonstrate competitive performance on vision-language tasks with faster inference speeds compared to Transformer-based models. Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference The paper, , presents a simplified Mamba-based architecture that addresses stability issues when scaling Mamba to larger sizes. The key innovation is EinFFT, a novel channel mixing technique that ensures stable optimization. SiMBA shows strong results on vision tasks and multivariate time series forecasting, closing the gap with state-of-the-art Transformers. SiMBA: Simplified Mamba-based Architecture for Vision and Multivariate Time series These three papers highlight the architectural flexibility and potential of the Mamba model, which is promising for future advancements in context window size and data type support. If you want to move beyond Transformers, that might be the way to go. You can find the Mamba repository here: https://github.com/state-spaces/mamba News From The Usual Suspects © https://x.com/emollick/status/1770289508589707625?s=20&embedable=true Microsoft Is Hungry Microsoft basically owns OpenAI ( — from Elon Musk’s against OpenAI). “Indeed, as the November 2023 drama was unfolding, Microsoft’s CEO boasted that it would not matter “[i]f OpenAI disappeared tomorrow.” He explained that “[w]e have all the IP rights and all the capability.” “We have the people, we have the compute, we have the data, we have everything.” “We are below them, above them, around them.” lawsuit Last February, Microsoft in Mistral and brought its newest AI model, Mistral Large, to Azure. invested Last week, Microsoft basically Inflection AI’s team, including two of its co-founders: Mustafa Suleyman and Karén Simonyan. Eric Newcomer at Microsoft’s creativity in non-acquisition strategies, such as partnering with and investing in companies like Inflection and OpenAI without formal acquisitions, sidestepping antitrust reviews. The Stratechery why many aspects of Inflection and its rapid ascent to Unicorn status were odd from the very beginning. Soma Somasegar from Madrona VC that Microsoft partners with Inflection.ai to innovate AI for consumers, aiming to transform its presence in the consumer market. acqui-hired points out explains thinks Stability AI Evokes Many Jokes About Instability This January, we published with the subtitle: “Investigating the Thin Line Between Genuine Innovation and Strategic Exaggeration in AI”. On March 23, after too many exaggerations Emad Mostaque from his position as CEO of Stability AI. According to him, he to concentrate on decentralized AI. What does it mean exactly — no one knows. Stability AI profile resigns wants OpenAI https://x.com/gdb/status/1772334028009652415?s=20&embedable=true Hugging Face announces the release of , the largest public domain dataset designed for training (LLMs. Encompassing 500 billion words across multiple languages including English, French, Dutch, Spanish, German, and Italian, it’s a multilingual collection sourced from diverse cultural heritage initiatives. This release aims to demonstrate the feasibility of developing open LLMs using copyright-free materials, supported by an international collaboration with organizations dedicated to open science in AI. Common Corpus Sam Altman Is Open-Sourcing His Orb The Worldcoin Foundation the Orb’s software (Orb is a device that scans irises to create unique digital IDs) open-source, aiming to enhance privacy and security in proving humanness online. The software, available under MIT/Apache 2.0 licenses on GitHub, supports World ID verification by processing images locally and ensuring data privacy through secure transfers. has made Enjoyed This Story? I write a weekly analysis of the AI world in the newsletter. We aim to equip you with comprehensive knowledge and historical insights so you can make informed decisions about AI and ML. Turing Post

This story contains new, firsthand information uncovered by the writer.

Hot off the press! This story contains factual information about a recent event.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Mamba Architecture: What Is It and Can It Beat Transformers?

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

AI Industries Converge: Llama 3 and Electric Atlas Have More In Common Than You Think

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

AI Industries Converge: Llama 3 and Electric Atlas Have More In Common Than You Think

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps