You know the feeling. You’re deep into a coding session with ChatGPT or Claude, pasting in your entire project structure, and suddenly—it forgets. The variable you defined 50 messages ago? Gone. The specific constraint you set for the database schema? Hallucinated into oblivion. For years, we’ve been dealing with the "Goldfish Problem" in AI. Transformers, for all their brilliance, have a hard limit: the context window. Even with 1M+ token windows, the cost is astronomical, and the recall gets fuzzy. "Goldfish Problem" But Google Research just dropped a bombshell that might finally kill the context bottleneck. It’s called Titans, and it’s paired with a new theoretical framework called MIRAS. Titans MIRAS Here is why this isn't just another paper—it’s the blueprint for the next generation of AGI. The Problem: Why Transformers Are Expensive Professors To understand why Titans is a big deal, we have to look at why current LLMs struggle with long-term memory. Think of a Transformer model (like GPT-4) as a brilliant professor. Before every single class, this professor has to re-read the entire textbook (your chat history) from scratch to answer your question. brilliant professor It’s Slow: Reading the whole book every time takes immense compute.
It’s Expensive: In technical terms, Attention mechanisms scale quadratically ($O(N^2)$). Double the input length, and you quadruple the cost.
It’s Static: Once the model is trained, its weights are frozen. It doesn't "learn" from your conversation; it just "attends" to the text you paste in the window. It’s Slow: Reading the whole book every time takes immense compute. It’s Slow: It’s Expensive: In technical terms, Attention mechanisms scale quadratically ($O(N^2)$). Double the input length, and you quadruple the cost. It’s Expensive: quadratically It’s Static: Once the model is trained, its weights are frozen. It doesn't "learn" from your conversation; it just "attends" to the text you paste in the window. It’s Static: We’ve tried fixing this with RNNs (Recurrent Neural Networks) and State Space Models (like Mamba), which compress memory into a fixed-size box. But that’s like asking the professor to summarize the whole textbook onto a single sticky note. You lose the details. RNNs State Space Models Enter Titans. Titans ![Image Description: A diagram comparing Transformer architecture (showing quadratic attention lines connecting every token to every other token) vs. Titans architecture (showing a linear flow with a separate 'Deep Memory' module alongside).] (Caption: Transformers read the whole history every time. Titans just update their memory.) Enter the Titans: The "Genius Friend" If a Transformer is a professor who re-reads the book, Titans is a genius friend who learns alongside you and never forgets. Titans is a genius friend who learns alongside you and never forgets. Titans introduces a Neural Long-Term Memory Module. Instead of compressing history into a small, fixed vector (like a sticky note), it uses a deep neural network (a Multi-Layer Perceptron) as the memory. Neural Long-Term Memory Module as Crucially, Titans learns at test time. Titans learns at test time. This is the viral hook: The model actually updates its own weights while you are talking to it. It doesn't just "store" data; it trains itself on your specific context in real-time. The model actually updates its own weights while you are talking to it. trains The "Surprise" Metric: How It Decides What to Remember How does it know what to keep and what to delete? It uses something remarkably human: Surprise. Surprise. Google researchers realized that humans don't remember everything. You don't remember putting on your socks this morning because it was routine (Low Surprise). But if you saw a zebra in your kitchen, you’d remember it forever (High Surprise). Titans uses gradients as a "Surprise Metric": Low Surprise: "The user typed const a = 1. Boring. I predicted that. Ignore." -> Gradient is small, memory barely updates.
High Surprise: "The user just pasted a massive, contradictory bug report that breaks my previous assumption." -> Gradient is huge, memory updates aggressively. Low Surprise: "The user typed const a = 1. Boring. I predicted that. Ignore." -> Gradient is small, memory barely updates. Low Surprise: const a = 1 Gradient is small, memory barely updates. High Surprise: "The user just pasted a massive, contradictory bug report that breaks my previous assumption." -> Gradient is huge, memory updates aggressively. High Surprise: Gradient is huge, memory updates aggressively. This allows Titans to process 2 million+ tokens (and theoretically infinite streams) without getting slower or dumber. 2 million+ tokens ![Image Description: A conceptual illustration of the 'Surprise Metric'. A brain filtering out grey 'routine' blocks but catching a bright red 'surprise' block and locking it into a vault.] (Caption: Titans only memorizes the 'Surprise' moments—just like your brain.) MIRAS: The Grand Unification Theory Titans is the tool, but MIRAS is the blueprint. tool MIRAS blueprint In the paper, Google didn't just drop a new model; they dropped a framework that unifies everything we know about sequence modeling. MIRAS stands for Memory, Interest (Attentional Bias), Retention, And Sequence optimization. M I R A S They argue that every successful model—from the 90s LSTM to the modern Transformer—is just a different flavor of Associative Memory. every Associative Memory MIRAS breaks memory down into four knobs you can turn: Memory Architecture: Where do we store info? (Vector? Matrix? Neural Network?)
Attentional Bias: What do we pay attention to?
Retention Gate: How fast do we forget? (The "Weight Decay" knob).
Memory Algorithm: How do we update the memory? (Gradient Descent? Hebbian learning?) Memory Architecture: Where do we store info? (Vector? Matrix? Neural Network?) Memory Architecture: Attentional Bias: What do we pay attention to? Attentional Bias: Retention Gate: How fast do we forget? (The "Weight Decay" knob). Retention Gate: Memory Algorithm: How do we update the memory? (Gradient Descent? Hebbian learning?) Memory Algorithm: By tweaking these knobs, Google created three new MIRAS variants—YAAD, MONETA, and MEMORA—each optimized for different kinds of stability and robustness. YAAD MONETA MEMORA "Instead of compressing information into a static state, this architecture actively learns and updates its own parameters as data streams in." — Google Research Blog "Instead of compressing information into a static state, this architecture actively learns and updates its own parameters as data streams in." — Google Research Blog "Instead of compressing information into a static state, this architecture actively learns and updates its own parameters as data streams in." Why This Matters (The "So What?") You might be thinking, "Cool, another paper. When can I use it?" Here is why you should care: This is the path to Persistent Agents. This is the path to Persistent Agents. Right now, "AI Agents" are kind of a lie. They are just scripts that chain prompts together. If an agent runs for 3 days, it eventually runs out of context space or gets confused. A Titan-based agent could theoretically run forever. forever Personalization: It remembers your preferences from 6 months ago because they are "baked" into its long-term neural memory.
Codebases: It could ingest a 10-million-line repo and actually "learn" the architecture, not just RAG-search for snippets.
Efficiency: It scales linearly. Processing a 1-hour video or a 100-hour video doesn't exponentially explode the cost. Personalization: It remembers your preferences from 6 months ago because they are "baked" into its long-term neural memory. Personalization: Codebases: It could ingest a 10-million-line repo and actually "learn" the architecture, not just RAG-search for snippets. Codebases: Efficiency: It scales linearly. Processing a 1-hour video or a 100-hour video doesn't exponentially explode the cost. Efficiency: linearly The Benchmark Killer In the BABILong benchmark (a brutal test of finding facts in massive documents), Titans crushed GPT-4 and other retrieval-based models. It didn't just "find" the needle in the haystack; it remembered where the needle was because it updated its internal state when it saw it. BABILong remembered Conclusion: The Memory Wall Has Fallen For the last five years, we’ve been building bigger and bigger brains (Parameters). But we’ve neglected the other half of intelligence: Memory. Memory. Titans proves that we don't need a 100-Trillion parameter model to remember a book. We need a model that processes information like we do: ignoring the noise, obsessing over the surprises, and learning in real-time. The era of the "Goldfish AI" is ending. The era of the Titan has begun. Titan 5 Takeaways for Developers: Context isn't enough: Simply making the context window bigger is a brute-force solution that hits a wall.
Test-Time Training is real: The idea of models updating weights during inference is no longer sci-fi.
Surprise = Information: If your data is predictable, your model shouldn't waste energy storing it.
MIRAS is the new meta: Expect future papers to frame their work using the MIRAS terminology.
Efficiency Wins: Linear scaling (Titans) will eventually beat Quadratic scaling (Transformers) for long-horizon tasks. Context isn't enough: Simply making the context window bigger is a brute-force solution that hits a wall. Context isn't enough: Test-Time Training is real: The idea of models updating weights during inference is no longer sci-fi. Test-Time Training is real: during Surprise = Information: If your data is predictable, your model shouldn't waste energy storing it. Surprise = Information: MIRAS is the new meta: Expect future papers to frame their work using the MIRAS terminology. MIRAS is the new meta: Efficiency Wins: Linear scaling (Titans) will eventually beat Quadratic scaling (Transformers) for long-horizon tasks. Efficiency Wins: Liked this breakdown? Smash that clap button and follow me for more deep dives into the papers changing our industry. Liked this breakdown? Smash that clap button and follow me for more deep dives into the papers changing our industry.

Google

The Goldfish Era is Over: How Google’s ‘Titans’ Gave AI Infinite Memory

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Google Just Killed LeetCode: Why AlphaEvolve is the End of the "Algorithm Interview"

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Google Just Killed LeetCode: Why AlphaEvolve is the End of the "Algorithm Interview"

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps