Your attention span has shrunk by nearly two minutes in the past 20 years, driven in part by technology’s hold over your work and home life, experts say. Here’s how to stay focused so you can get things done.

Your attention span is shrinking, studies say. Here’s how to stay focused

Cannes Film Festival officials are notoriously strict when it comes to enforcing protocols for black-tie ensembles on guests before they are allowed to ascend the iconic Palais steps. However, at Tuesday’s glamorous opening night ceremony, one guest managed to sneak by security on four legs ...

‘Anatomy of a Fall’ Breakout Messi the Dog Laps Up Attention at Cannes

Rebecca Joynes, 30, has pleaded not guilty to six charges of sexual activity with two teenagers while she was a teacher. She told a court she &#x27;liked&#x27; the attention but denies having sex with the boys

Teacher Rebecca Joynes says she was 'lonely and craved attention' and admits boy, 15, spent night at her flat - Mirror Online

A teacher denies the allegations of sexual activity with boys claiming she only wanted attention.

Salford: Teacher 'liked attention' but denies sex with boys

Rebecca Joynes denies four charges of sexual activity with two teenagers and two counts of sexual activity while being in position of trust

Teacher 'caved in' to attention from pupils but denies having sex whilst teaching them

Rebecca Joynes wept as she told jurors that her baby was taken away ‘24 hours after she was born’

Teacher accused of sex with children tells court she was ‘lonely and craved attention’ | The Independent

The Metropolitan Police has launched a murder probe in south London after the death of the 26-year-old died of stab wounds.Separate stabbing incidents » · Conservative leadership hopeful Boris Johnson has defended his infamous description of black people in Africa bearing “watermelon smiles” ...

'Lonely' teacher says she liked attention but denies having sex with boys

Rebecca Joynes, 30, denies any sexual activity took place, has pleaded not guilty to six charges of sexual activity with two teenagers and is on trial at Manchester Crown Court.

Teacher denies having sex with boys as she tells court she 'liked' attention after break-up | UK News | Sky News

Will Senate Democrats hold hearings into Donald Trump&#x27;s controversial pitch to oil industry executives? It&#x27;s “highly likely,” one key member said.

Trump’s pitch to Big Oil execs sparks Senate Democrats’ attention

The Blueshirts need to channel what they had the last time they ended the Hurricanes’ season, in Game 7 of their 2022 second-round series.

Rangers’ attention turns to Game 5 for second shot at Hurricanes knockout punch

attenio

Authors:
(1) Albert Gu, Machine Learning Department, Carnegie Mellon University;
(2)Tri Dao, Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me. Table of Links Abstract and 1. Introduction 2 State Space Models 3 Selective State Space Models and 3.1 Motivation: Selection as a Means of Compression 3.2 Improving SSMs with Selection 3.3 Efficient Implementation of Selective SSMs 3.4 A Simplifed SSM Architecture 3.5 Properties of Selection Mechanisms 3.6 Additional Model Details 4 Empirical Evaluation and 4.1 Synthetic Tasks 4.2 Language Modeling 4.3 DNA Modeling 4.4 Audio Modeling and Generation 4.5 Speed and Memory Benchmarks 4.6 Model Ablations 5 Discussion 6 Conclusion, Acknowledgments and References A Discussion: Selection Mechanism B Related Work and B.1 S4 Variants and Derivatives B.2 SSM Architectures B.3 Relationship to RNNs B.4 Linear Attention and B.5 Long Context Models C Mechanics of Selective SSMs D Hardware-aware Algorithm For Selective SSMs E Experimental Details and Additional Results and E.1 Synthetic Tasks E.2 Language Modeling E.3 DNA Modeling E.4 Audio Details E.5 Efficiency Benchmark B.4 Linear Attention The Linear Attention (LA) (Katharopoulos et al. 2020) framework is an important result popularizing kernel attention and showing how it relates to recurrent autoregressive models. Many variants have proposed alternative kernels and other modifications. Random Feature Attention (RFA) (H. Peng et al. 2021) chooses the kernel feature map to approximate softmax attention (i.e. the exp feature map) using the random Fourier feature approximation of Gaussian kernels (Rahimi and Recht 2007). Performer (Choromanski et al. 2021) finds an approximation to the exponential kernel involving only positive features, which also allows the softmax normalization term. TransNormer (Qin, Han, W. Sun, D. Li, et al. 2022) showed that the LA denominator term can be unstable and proposed replacing it with a LayerNorm. cosFormer (Qin, W. Sun, et al. 2022) augments RFA with a cosine reweighting mechanism that incorporates positional information to emphasize locality. Linear Randomized Attention (Zheng, C. Wang, and L. Kong 2022) generalize RFA from the perspective of importance sampling, and generalize it to provide better estimates of the full softmax kernel (rather than just the exp-transformed numerator). Aside from kernel attention, many other variants of efficient attention exist; the survey Tay, Dehghani, Bahri, et al. (2022) offers an extensive categorization of many of these. B.5 Long Context Models Long context has become a popular subject, and several recent models have claimed to scale to longer and longer sequences. However, these are often from a computational standpoint and have not been extensively validated. These include: In contrast, we believe this work presents one of the first approaches to meaningfully demonstrate increasing performance with longer context. C Mechanics of Selective SSMs The discretization step size is where we observe that the parameter can be viewed as a learnable bias and folded into the linear projection. Now applying the zero-order hold (ZOH) discretization formulas: Thus the final discrete recurrence (2a) is as desired. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Albert Gu, Machine Learning Department, Carnegie Mellon University; (2)Tri Dao, Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me. Authors: Authors: (1) Albert Gu, Machine Learning Department, Carnegie Mellon University; (2)Tri Dao, Department of Computer Science, Princeton University agu@cs.cmu.edu, tri@tridao.me. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 State Space Models 2 State Space Models 3 Selective State Space Models and 3.1 Motivation: Selection as a Means of Compression 3 Selective State Space Models and 3.1 Motivation: Selection as a Means of Compression 3.2 Improving SSMs with Selection 3.2 Improving SSMs with Selection 3.3 Efficient Implementation of Selective SSMs 3.3 Efficient Implementation of Selective SSMs 3.4 A Simplifed SSM Architecture 3.4 A Simplifed SSM Architecture 3.5 Properties of Selection Mechanisms 3.5 Properties of Selection Mechanisms 3.6 Additional Model Details 3.6 Additional Model Details 4 Empirical Evaluation and 4.1 Synthetic Tasks 4 Empirical Evaluation and 4.1 Synthetic Tasks 4.2 Language Modeling 4.2 Language Modeling 4.3 DNA Modeling 4.3 DNA Modeling 4.4 Audio Modeling and Generation 4.4 Audio Modeling and Generation 4.5 Speed and Memory Benchmarks 4.5 Speed and Memory Benchmarks 4.6 Model Ablations 4.6 Model Ablations 5 Discussion 5 Discussion 6 Conclusion, Acknowledgments and References 6 Conclusion, Acknowledgments and References A Discussion: Selection Mechanism A Discussion: Selection Mechanism B Related Work and B.1 S4 Variants and Derivatives B Related Work and B.1 S4 Variants and Derivatives B.2 SSM Architectures B.2 SSM Architectures B.3 Relationship to RNNs B.3 Relationship to RNNs B.4 Linear Attention and B.5 Long Context Models B.4 Linear Attention and B.5 Long Context Models C Mechanics of Selective SSMs C Mechanics of Selective SSMs D Hardware-aware Algorithm For Selective SSMs D Hardware-aware Algorithm For Selective SSMs E Experimental Details and Additional Results and E.1 Synthetic Tasks E Experimental Details and Additional Results and E.1 Synthetic Tasks E.2 Language Modeling E.2 Language Modeling E.3 DNA Modeling E.3 DNA Modeling E.4 Audio Details E.4 Audio Details E.5 Efficiency Benchmark E.5 Efficiency Benchmark B.4 Linear Attention The Linear Attention (LA) (Katharopoulos et al. 2020) framework is an important result popularizing kernel attention and showing how it relates to recurrent autoregressive models. Many variants have proposed alternative kernels and other modifications. Random Feature Attention (RFA) (H. Peng et al. 2021) chooses the kernel feature map to approximate softmax attention (i.e. the exp feature map) using the random Fourier feature approximation of Gaussian kernels (Rahimi and Recht 2007). Performer (Choromanski et al. 2021) finds an approximation to the exponential kernel involving only positive features, which also allows the softmax normalization term. TransNormer (Qin, Han, W. Sun, D. Li, et al. 2022) showed that the LA denominator term can be unstable and proposed replacing it with a LayerNorm. cosFormer (Qin, W. Sun, et al. 2022) augments RFA with a cosine reweighting mechanism that incorporates positional information to emphasize locality. Linear Randomized Attention (Zheng, C. Wang, and L. Kong 2022) generalize RFA from the perspective of importance sampling, and generalize it to provide better estimates of the full softmax kernel (rather than just the exp-transformed numerator). Aside from kernel attention, many other variants of efficient attention exist; the survey Tay, Dehghani, Bahri, et al. (2022) offers an extensive categorization of many of these. B.5 Long Context Models Long context has become a popular subject, and several recent models have claimed to scale to longer and longer sequences. However, these are often from a computational standpoint and have not been extensively validated. These include: In contrast, we believe this work presents one of the first approaches to meaningfully demonstrate increasing performance with longer context. C Mechanics of Selective SSMs The discretization step size is where we observe that the parameter can be viewed as a learnable bias and folded into the linear projection. Now applying the zero-order hold (ZOH) discretization formulas: Thus the final discrete recurrence (2a) is as desired. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Linear Attention and Long Context Models

MONROVIA, Liberia, February 28th, 2025/Chainwire/--Chad Leads the Way: AI-Powered Memecoins Are Evolving. Memeagent has introduced Chad, an autonomous AI agent designed to bring intelligence and automation to the memecoin ecosystem. While many meme projects rely on fleeting trends, Memeagent is built differently – a fully AI agent-driven ecosystem designed to change the game for good. As AI reshapes the landscape, Memeagent is positioning itself at the forefront of this evolution. Chad: The AI Engine Driving Memeagent’s Evolution Chad is a key figure who represents Memeagent. He’s a fully automated AI agent that helps users to navigate Memeagent. Memeagent is an AI-powered ecosystem designed for sustained growth and real utility. At the core of this innovation is Chad, Memeagent’s flagship AI agent, showcasing the advanced capabilities that will soon be available to all users. Built on ElizaOS, a cutting-edge open-source framework for autonomous AI agents, Chad can think, learn, and interact independently. With persistent memory, adaptability, and live data integration, he stays ahead of trends, making real-time decisions in an ever-evolving market. ElizaOS is a pioneer in decentralized AI (DeAI), allowing Chad to operate seamlessly within blockchain ecosystems. With features like encrypted transactions, asset management, and real-time analytics, Chad isn’t just generating memes—he’s actively shaping market dynamics. Working 24/7, Chad monitors trends, predicts market shifts, and engages across major social platforms without hesitation. Unlike human traders, he never sleeps, never FOMOs, and never misses an opportunity – driving Memeagent’s AI-driven viral growth. He will automatically respond to all users needs and help them navigate through Memeagent itself. The Reasons Memeagent Is Gaining Attention Behind the scenes, big moves are happening. Memeagent combines AI-powered engagement, predictive analytics, and automated viral marketing into one platform, making it one of the most powerful ecosystem for meme adoption yet. Here’s why serious players are locking in: Always-On AI – Chad works around the clock, monitoring X/Twitter, Telegram, Instagram, and more to maximize exposure.
Predictive Trend Analysis – Spot what’s about to go viral before the rest of the market catches on.
Automated Growth – Hands-free content and engagement across major social platforms.
No Technical Barriers – AI handles the heavy lifting, so anyone can capitalize. $MA: The Fuel Behind the Memeagent Ecosystem Unlike other meme coins, $MA goes beyond speculation—it powers the entire Memeagent platform, unlocking a range of utilities that set it apart. AI agent Creation – Use $MA to create and deploy custom AI-powered Memeagents that generate viral content and interact across social platforms.
DeFi Utilities – Stake $MA for passive income and participate in governance decisions to shape the future of the Memeagent ecosystem.
Deflationary Impact – A portion of every $MA transaction is burned, reducing supply and increasing scarcity over time. Memeagent and Chad Gain Traction as AI-Powered Platforms Expand According to the team, Memeagent isn’t another speculative meme coin—it’s a structured, AI-powered ecosystem with long-term incentives. The Features That Set Memeagent Apart AI-Driven Growth – Automation supports long-term engagement beyond short-term trends.
$MA Token Utility – Powers AI tools, governance, staking, and a deflationary tokenomics model.
Security & Infrastructure – Built with SOC 2 Type 2-certified security for reliability and trust. Memeagent’s Roadmap is Packed Memeagent is gaining attention as its ecosystem expands. With a structured roadmap ahead and continuous development, the platform is set for steady growth. Upcoming milestones: Q2 2025: Beta AI automation tools roll out to early users.
Q3 2025: Advanced predictive analytics and expanded AI engagement.
Q4 2025: Full staking launch, multilingual support, and deeper integrations.
2026: Memeagent’s ecosystem expands to full-scale adoption. Market Movers Are Taking Notice Memeagent is already in motion, leveraging AI and automation to drive engagement. As the ecosystem expands, those recognizing its potential are positioning themselves early. The Next Evolution of Memecoins Is Built on Innovation, Not Just Hype. AI-Powered Innovation in Memecoins Memeagent is introducing AI-driven engagement to the memecoin ecosystem. As the platform grows, early users can explore $MA’s utility and experience Chad’s AI-powered automation in action. Users can visit www.memeagent.io to get started now. #CHADISCOMING About Memeagent Memeagent is developed by a team of experienced AI engineers and blockchain specialists with extensive backgrounds in DeFi, yield farming, token design, and AI-driven automation. Active in the crypto space since 2016, the team has contributed to various high-profile projects, focusing on scalability, security, and decentralized AI (DeAI) solutions. Contact Benjamin Peng info@memeagent.io This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here MONROVIA, Liberia, February 28th, 2025/Chainwire/--Chad Leads the Way: AI-Powered Memecoins Are Evolving. Memeagent has introduced Chad, an autonomous AI agent designed to bring intelligence and automation to the memecoin ecosystem. While many meme projects rely on fleeting trends, Memeagent is built differently – a fully AI agent-driven ecosystem designed to change the game for good. As AI reshapes the landscape, Memeagent is positioning itself at the forefront of this evolution. Chad: The AI Engine Driving Memeagent’s Evolution Chad is a key figure who represents Memeagent. He’s a fully automated AI agent that helps users to navigate Memeagent. Memeagent is an AI-powered ecosystem designed for sustained growth and real utility. At the core of this innovation is Chad, Memeagent’s flagship AI agent, showcasing the advanced capabilities that will soon be available to all users. Built on ElizaOS, a cutting-edge open-source framework for autonomous AI agents, Chad can think, learn, and interact independently. With persistent memory, adaptability, and live data integration, he stays ahead of trends, making real-time decisions in an ever-evolving market. ElizaOS is a pioneer in decentralized AI (DeAI), allowing Chad to operate seamlessly within blockchain ecosystems. With features like encrypted transactions, asset management, and real-time analytics, Chad isn’t just generating memes—he’s actively shaping market dynamics. Working 24/7, Chad monitors trends, predicts market shifts, and engages across major social platforms without hesitation. Unlike human traders, he never sleeps, never FOMOs, and never misses an opportunity – driving Memeagent’s AI-driven viral growth. He will automatically respond to all users needs and help them navigate through Memeagent itself. The Reasons Memeagent Is Gaining Attention Behind the scenes, big moves are happening. Memeagent combines AI-powered engagement, predictive analytics, and automated viral marketing into one platform, making it one of the most powerful ecosystem for meme adoption yet. Here’s why serious players are locking in: Always-On AI – Chad works around the clock, monitoring X/Twitter, Telegram, Instagram, and more to maximize exposure. Predictive Trend Analysis – Spot what’s about to go viral before the rest of the market catches on. Automated Growth – Hands-free content and engagement across major social platforms. No Technical Barriers – AI handles the heavy lifting, so anyone can capitalize. Always-On AI – Chad works around the clock, monitoring X/Twitter, Telegram, Instagram, and more to maximize exposure. Predictive Trend Analysis – Spot what’s about to go viral before the rest of the market catches on. Automated Growth – Hands-free content and engagement across major social platforms. No Technical Barriers – AI handles the heavy lifting, so anyone can capitalize. $MA: The Fuel Behind the Memeagent Ecosystem Unlike other meme coins, $MA goes beyond speculation—it powers the entire Memeagent platform, unlocking a range of utilities that set it apart. AI agent Creation – Use $MA to create and deploy custom AI-powered Memeagents that generate viral content and interact across social platforms. DeFi Utilities – Stake $MA for passive income and participate in governance decisions to shape the future of the Memeagent ecosystem. Deflationary Impact – A portion of every $MA transaction is burned, reducing supply and increasing scarcity over time. AI agent Creation – Use $MA to create and deploy custom AI-powered Memeagents that generate viral content and interact across social platforms. DeFi Utilities – Stake $MA for passive income and participate in governance decisions to shape the future of the Memeagent ecosystem. Deflationary Impact – A portion of every $MA transaction is burned, reducing supply and increasing scarcity over time. Memeagent and Chad Gain Traction as AI-Powered Platforms Expand According to the team, Memeagent isn’t another speculative meme coin—it’s a structured, AI-powered ecosystem with long-term incentives. The Features That Set Memeagent Apart AI-Driven Growth – Automation supports long-term engagement beyond short-term trends. $MA Token Utility – Powers AI tools, governance, staking, and a deflationary tokenomics model. Security & Infrastructure – Built with SOC 2 Type 2-certified security for reliability and trust. AI-Driven Growth – Automation supports long-term engagement beyond short-term trends. $MA Token Utility – Powers AI tools, governance, staking, and a deflationary tokenomics model. Security & Infrastructure – Built with SOC 2 Type 2-certified security for reliability and trust. Memeagent’s Roadmap is Packed Memeagent is gaining attention as its ecosystem expands. With a structured roadmap ahead and continuous development, the platform is set for steady growth. Upcoming milestones: Q2 2025: Beta AI automation tools roll out to early users. Q3 2025: Advanced predictive analytics and expanded AI engagement. Q4 2025: Full staking launch, multilingual support, and deeper integrations. 2026: Memeagent’s ecosystem expands to full-scale adoption. Q2 2025: Beta AI automation tools roll out to early users. Q3 2025: Advanced predictive analytics and expanded AI engagement. Q4 2025: Full staking launch, multilingual support, and deeper integrations. 2026: Memeagent’s ecosystem expands to full-scale adoption. Market Movers Are Taking Notice Memeagent is already in motion, leveraging AI and automation to drive engagement. As the ecosystem expands, those recognizing its potential are positioning themselves early. The Next Evolution of Memecoins Is Built on Innovation, Not Just Hype. AI-Powered Innovation in Memecoins Memeagent is introducing AI-driven engagement to the memecoin ecosystem. As the platform grows, early users can explore $MA’s utility and experience Chad’s AI-powered automation in action. Users can visit www.memeagent.io to get started now. www.memeagent.io www.memeagent.io #CHADISCOMING About Memeagent Memeagent is developed by a team of experienced AI engineers and blockchain specialists with extensive backgrounds in DeFi, yield farming, token design, and AI-driven automation. Active in the crypto space since 2016, the team has contributed to various high-profile projects, focusing on scalability, security, and decentralized AI (DeAI) solutions. Memeagent Memeagent Contact Benjamin Peng info@memeagent.io This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here here here

Memeagent Is Gaining Attention – Meeting Chad, The AI Agent That’s Changing The Game

Memeagent está ganando atención: conozca a Chad, el agente de inteligencia artificial que está cambiando las reglas del juego

Memeagent está a chamar a atención: coñece a Chad, o axente de IA que está a cambiar o xogo

Memeagent privlači pažnju – upoznajte Chada, AI agenta koji mijenja igru

Memeagent が注目を集める – ゲームを変える AI エージェント Chad との出会い

Memeagent добива внимание - средба со Чад, агентот за вештачка интелигенција што ја менува играта

Memeagent E Hwetša Tlhokomelo – Go Kopana le Chad, Moemedi wa AI Yo a Fetošago Papadi

Memeagent atrage atenția – Întâlnesc cu Chad, agentul AI care schimbă jocul

Memeagent таваҷҷӯҳро ба худ ҷалб мекунад - Вохӯрӣ бо Чад, агенти AI, ки бозиро тағир медиҳад

Memeagent Is Gaining Attention – Meeting Chad, Ang AI Agent That’s Changing The Game

Memeagent Dikkat Çekiyor – Oyunu Değiştiren Yapay Zeka Ajanı Chad ile Tanışma

Memeagent Yi Kuma Nyingiso – Ku Hlangana na Chad, Muyimeri wa AI Loyi A Cincaka Ntlangu

Memeagent đang thu hút sự chú ý – Gặp gỡ Chad, đặc vụ AI đang thay đổi trò chơi

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques D. Multi-Group Attention Family D.1. Detailed Analysis on Memory Access We show in Table 4 that the memory IO cost for ⟨q, K⟩ is dominated by the loading of K which costs bmhk in the case of multihead where g = h. This cost is particularly high due to the coupling of batch size b, context length m, and the entire hidden dimension d. Compared to the number of computations, which has complexity bmd, this attention module requires one memory IO per one tensor operation (memory-io bound). In contrast, other operations such as feedforw can be the main bottleneck for incremental decoding and our paper aims to tackle such problems.ard has much lower ratio of memory IO per compute (compute bound). These attention computation D.2. Model FLOPs The scaling laws by Kaplan et al. (2020) shows that the modelrelated FLOPs during the forward pass is 2N where N is the number of parameters (without the embeddings). We show that it holds for a general multi-group model as well. The only difference between the multi-group and the multi-head case is the projection PK and PV where they are of size dgk instead of dhk. Since this is a linear layer, the forward pass FLOPs for any input is still proportional such projection size. Therefore, it follows that for any multi-group attention, including multi-head, the forward FLOPs is 2N where N is the respective number of parameters. D.3. Comparing Capabilities-Equivalent Models This section outlines the analysis of latency change when we switch from an MH model to an MG model with F times the size. D.3.1. CONTEXT ENCODING D.3.2. INCREMENTAL DECODING This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques D. Multi-Group Attention Family D.1. Detailed Analysis on Memory Access We show in Table 4 that the memory IO cost for ⟨q, K⟩ is dominated by the loading of K which costs bmhk in the case of multihead where g = h. This cost is particularly high due to the coupling of batch size b, context length m, and the entire hidden dimension d. Compared to the number of computations, which has complexity bmd, this attention module requires one memory IO per one tensor operation (memory-io bound). In contrast, other operations such as feedforw can be the main bottleneck for incremental decoding and our paper aims to tackle such problems.ard has much lower ratio of memory IO per compute (compute bound). These attention computation D.2. Model FLOPs The scaling laws by Kaplan et al. (2020) shows that the modelrelated FLOPs during the forward pass is 2N where N is the number of parameters (without the embeddings). We show that it holds for a general multi-group model as well. The only difference between the multi-group and the multi-head case is the projection PK and PV where they are of size dgk instead of dhk. Since this is a linear layer, the forward pass FLOPs for any input is still proportional such projection size. Therefore, it follows that for any multi-group attention, including multi-head, the forward FLOPs is 2N where N is the respective number of parameters. D.3. Comparing Capabilities-Equivalent Models This section outlines the analysis of latency change when we switch from an MH model to an MG model with F times the size. D.3.1. CONTEXT ENCODING D.3.1. CONTEXT ENCODING D.3.2. INCREMENTAL DECODING D.3.2. INCREMENTAL DECODING This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Understanding Multi-Group Attention in AI Models

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques A. FAQs Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary?
A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts.

Q: How applicable is multi-query for single-batch inference without high batch sampling?
A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1.

Q: Is bifurcated attention applicable for the case where we process different inputs in a batch?
A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper.

Q: Any caveats to using bifurcated attention?
A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency.

Q: How does model quantization (or lower precision arithmetic) affect the findings?
A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same.

Q: Does the conclusion depend on the inference implementation or different hardware?
A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis.

Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques A. FAQs Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary?
A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts. Q: How applicable is multi-query for single-batch inference without high batch sampling?
A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1. Q: Is bifurcated attention applicable for the case where we process different inputs in a batch?
A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper. Q: Any caveats to using bifurcated attention?
A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency. Q: How does model quantization (or lower precision arithmetic) affect the findings?
A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same. Q: Does the conclusion depend on the inference implementation or different hardware?
A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis. Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary? A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts. Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary? Q: A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts. A: Q: How applicable is multi-query for single-batch inference without high batch sampling? A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1. Q: How applicable is multi-query for single-batch inference without high batch sampling? Q: A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1. A: Q: Is bifurcated attention applicable for the case where we process different inputs in a batch? A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper. Q: Is bifurcated attention applicable for the case where we process different inputs in a batch? Q: A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper. A: Q: Any caveats to using bifurcated attention? A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency. Q : Any caveats to using bifurcated attention? Q A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency. A: Q: How does model quantization (or lower precision arithmetic) affect the findings? A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same. Q: How does model quantization (or lower precision arithmetic) affect the findings? Q: A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same. A: Q: Does the conclusion depend on the inference implementation or different hardware? A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis. Q: Does the conclusion depend on the inference implementation or different hardware? A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis. Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? Q: This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

FAQs: How Bifurcated Attention Improves AI Model Efficiency

Faster AI Code Generation with Bifurcated Attention

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques 5. Experiments We first conduct experiments to see how capabilities scale with respect to model size for each attention type in Section 5.1. We find that attention types with higher compression (lower number of attention groups g) require model size compensation, ≈ 10% for multi-query (g = 1). We use such findings to compare the latency between the multi-head and the larger multi-query models of equal capabilities in Section 5.2. In Section 5.2.2, we focus on the single-context batch sampling scenario where we demonstrate the significant latency reduction of bifurcated attention and revisit the comparison between multi-head and multi-query in light of bifurcated attention. We outline inference details in Appendix C.5. 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention To compare the capabilities of different attention mechanisms, one can either scale other model configurations such as the number of layers ℓ, the number of heads h in order to make match the total model sizes between different attentions. However, it is often difficult to match the number of parameters exactly. In this work, we compare different attention mechanisms via the loss-vs-size scaling laws. For the setup, we use the model hyperparameters similar to that of GPT-3, where the size ranges from 125M to 13B, with hyperparameters such as ℓ, h, k increasing in tandem. Then, we consider three cases where g = 1 (multi-query), g = h (multi-head) and 1 < g < h (multi-group) where Appendix C.1 and C.2 shows the training and model configuration details. We train all three attention models of each size and plot the validation loss versus model size, shown in Figure 3. Our findings are summarized below. Higher number of attention groups g leads to higher expressiveness The results in Figure 3 shows the validation loss versus model size (log scale). The results indicate that, for the same model size (vertical slice across the plot), multi-head attention g = h achieves the lowest validation loss compared to 1 < g < h (multi-group) and g = 1 (multiquery). This trend holds consistently over three orders of magnitude of model sizes, where the curves corresponding to multi-head, multi-group and multi-query do not cross, implying that the rank of model expressiveness, or relative capabilities per number of parameters, is quite stable. An intuitive explanation is that the lower g corresponds to a lower rank representation of the key and value tensors, which encodes lower representation power of the past context and therefore yields lower capabilities than higher g, given the same model size. Scaling laws via downstream performance We use the average scores from two code generation benchmarks, multilingual HumanEval and MBXP (Athiwaratkun et al., 2022), as a proxy for model capabilities in addition to the validation loss. This approach is similar to that of the GPT-4 technical report (OpenAI, 2023) where HumanEval (Python) (Chen et al., 2021) is used to track the performance across multiple magnitudes of compute. In our case, we average across all 13 evaluation languages and two benchmarks to obtain a more stable proxy for capabilities. The result in Figure 3 demonstrates similar trend compared to the validation loss where the pass rate curves indicate the same relative expressiveness for multi-head, multi-group and multi-query attention. Matching capabilities by model size compensation Given the same capabilities (horizontal slice of the plot in Figure 3), the distance between two curves indicates the model size difference that the lower-rank attention needs to compensate in order to match the multi-head model performance. Empirically, we average the distance along the interpolated lines (log scale) and find this to correspond to 1.104 times; that is, a multi-query model can have the same capabilities as the multi-head model if the size is increased by ≈ 10% of the multi-head model size. Similarly, the gap is < 10% for multi-group attention. Alternatively, one can argue that a multi-query model of the same size could match a multi-head if the multi-query model is given more compute. However, in the regime where we train language models until or close to convergence and the performance saturates with respect to compute, the difference in capabilities will likely remain. Therefore, the size compensation is likely the most fair approach for comparison. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques 5. Experiments We first conduct experiments to see how capabilities scale with respect to model size for each attention type in Section 5.1. We find that attention types with higher compression (lower number of attention groups g) require model size compensation, ≈ 10% for multi-query (g = 1). We use such findings to compare the latency between the multi-head and the larger multi-query models of equal capabilities in Section 5.2. In Section 5.2.2, we focus on the single-context batch sampling scenario where we demonstrate the significant latency reduction of bifurcated attention and revisit the comparison between multi-head and multi-query in light of bifurcated attention. We outline inference details in Appendix C.5. 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention To compare the capabilities of different attention mechanisms, one can either scale other model configurations such as the number of layers ℓ, the number of heads h in order to make match the total model sizes between different attentions. However, it is often difficult to match the number of parameters exactly. In this work, we compare different attention mechanisms via the loss-vs-size scaling laws. For the setup, we use the model hyperparameters similar to that of GPT-3, where the size ranges from 125M to 13B, with hyperparameters such as ℓ, h, k increasing in tandem. Then, we consider three cases where g = 1 (multi-query), g = h (multi-head) and 1 < g < h (multi-group) where Appendix C.1 and C.2 shows the training and model configuration details. We train all three attention models of each size and plot the validation loss versus model size, shown in Figure 3. Our findings are summarized below. Higher number of attention groups g leads to higher expressiveness The results in Figure 3 shows the validation loss versus model size (log scale). The results indicate that, for the same model size (vertical slice across the plot), multi-head attention g = h achieves the lowest validation loss compared to 1 < g < h (multi-group) and g = 1 (multiquery). This trend holds consistently over three orders of magnitude of model sizes, where the curves corresponding to multi-head, multi-group and multi-query do not cross, implying that the rank of model expressiveness, or relative capabilities per number of parameters, is quite stable. An intuitive explanation is that the lower g corresponds to a lower rank representation of the key and value tensors, which encodes lower representation power of the past context and therefore yields lower capabilities than higher g, given the same model size. Higher number of attention groups leads to higher expressiveness Scaling laws via downstream performance We use the average scores from two code generation benchmarks, multilingual HumanEval and MBXP (Athiwaratkun et al., 2022), as a proxy for model capabilities in addition to the validation loss. This approach is similar to that of the GPT-4 technical report (OpenAI, 2023) where HumanEval (Python) (Chen et al., 2021) is used to track the performance across multiple magnitudes of compute. In our case, we average across all 13 evaluation languages and two benchmarks to obtain a more stable proxy for capabilities. The result in Figure 3 demonstrates similar trend compared to the validation loss where the pass rate curves indicate the same relative expressiveness for multi-head, multi-group and multi-query attention. Scaling laws via downstream performance Matching capabilities by model size compensation Given the same capabilities (horizontal slice of the plot in Figure 3), the distance between two curves indicates the model size difference that the lower-rank attention needs to compensate in order to match the multi-head model performance. Empirically, we average the distance along the interpolated lines (log scale) and find this to correspond to 1.104 times; that is, a multi-query model can have the same capabilities as the multi-head model if the size is increased Matching capabilities by model size compensation by ≈ 10% of the multi-head model size. Similarly, the gap is < 10% for multi-group attention. Alternatively, one can argue that a multi-query model of the same size could match a multi-head if the multi-query model is given more compute. However, in the regime where we train language models until or close to convergence and the performance saturates with respect to compute, the difference in capabilities will likely remain. Therefore, the size compensation is likely the most fair approach for comparison. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

A Little Less Memory, A Lot More Speed: The Science of Bifurcated Attention

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention Multi-query attention, proposed by Shazeer (2019), is an attention mechanism for transformers models that uses a single head for the key and value tensors, compared to h heads in the traditional multi-head attention (Vaswani et al., 2017). This technique effectively reduces the KV memory IO by h times, which leads to higher inference efficiency during incremental decoding. In effect, the single-head key or value tensor is shared and used to attend to all the multi-head query, hence the name multi-query. This corresponds to a compression in representation power of the key and value tensor, which we will see in the scaling laws study (Section 5.1) that it results in a reduced expressiveness in terms of model parameter efficiency. Such reduced expressiveness can be compensated by scaling the model bigger than the multi-head counterpart to match the representation power. We can also extrapolate these insights to a generalized multiquery attention mechanism (Ainslie et al., 2023), which provides a framework to understand both multi-query and multi-head attention, and everything in between. Here, the degree of KV compression is dictated by the number of attention groups g, where we alternatively refer to the generalized multi-query as multi-group. Each attention group can be interpreted as the broadcasted attention between a single head of key or value tensor, and multiple heads of query. In this paradigm, multi-query attention is a special case where the number of groups g = 1; that is, there is exactly one such group. Conversely, multi-head attention is another special case where the number of attention groups matches the number of heads (g = h), in which case each head in the key or value tensor attends to one head in the query. More generally, the number of groups g can lie anywhere between 1 and h, indicating various degrees of compression. For practical purposes, it is most convenient when g divides h. The attention mechanism in this setting can be expressed in terms of Einstein summation as: logits = ⟨q, K⟩ : einsum(bgpnk, bgmk) → bgpnm (1) o = ⟨w, V ⟩ : einsum(bgpmn, bgmv) → bgpnv (2) This generalized multi-group attention mechanism thus provides a unified perspective on the design space of attention architectures. By adjusting the number of attention groups g, one can flexibly tune these trade-offs, potentially yielding new regimes of performance for transformer models. In Section 5.1, we will look into such capability vs latency trade-off. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention Multi-query attention, proposed by Shazeer (2019), is an attention mechanism for transformers models that uses a single head for the key and value tensors, compared to h heads in the traditional multi-head attention (Vaswani et al., 2017). This technique effectively reduces the KV memory IO by h times, which leads to higher inference efficiency during incremental decoding. In effect, the single-head key or value tensor is shared and used to attend to all the multi-head query, hence the name multi-query. This corresponds to a compression in representation power of the key and value tensor, which we will see in the scaling laws study (Section 5.1) that it results in a reduced expressiveness in terms of model parameter efficiency. Such reduced expressiveness can be compensated by scaling the model bigger than the multi-head counterpart to match the representation power. We can also extrapolate these insights to a generalized multiquery attention mechanism (Ainslie et al., 2023), which provides a framework to understand both multi-query and multi-head attention, and everything in between. Here, the degree of KV compression is dictated by the number of attention groups g, where we alternatively refer to the generalized multi-query as multi-group. Each attention group can be interpreted as the broadcasted attention between a single head of key or value tensor, and multiple heads of query. In this paradigm, multi-query attention is a special case where the number of groups g = 1; that is, there is exactly one such group. Conversely, multi-head attention is another special case where the number of attention groups matches the number of heads (g = h), in which case each head in the key or value tensor attends to one head in the query. More generally, the number of groups g can lie anywhere between 1 and h, indicating various degrees of compression. For practical purposes, it is most convenient when g divides h. The attention mechanism in this setting can be expressed in terms of Einstein summation as: logits = ⟨q, K⟩ : einsum(bgpnk, bgmk) → bgpnm (1) o = ⟨w, V ⟩ : einsum(bgpmn, bgmv) → bgpnv (2) This generalized multi-group attention mechanism thus provides a unified perspective on the design space of attention architectures. By adjusting the number of attention groups g, one can flexibly tune these trade-offs, potentially yielding new regimes of performance for transformer models. In Section 5.1, we will look into such capability vs latency trade-off. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Why Multi-Query Attention Matters for Large Language Models

Hackers are particularly good at one thing: Hacking. But hackers are particularly bad at another thing: Sports. Hackers who invest physical effort into their well-being are the minority. They use their bodies to perform organic duties, everything else falls into a clearly defined role model between brain and brain-container. The image of a pale, over-sugared and anti-social loner who invents Bitcoin at night in a disused factory in front of a fap station with 16 screens corresponds to the hacker’s ideal, but one thing usually becomes a problem with the hacker’s ideal: That the goal is to make the structure-porn that circulates in the hacker’s brain a reality, but structure-porn rarely succeeds on the first attempt. For structure-porn to get delivered successfully, many manual repetitions are necessary, and hackers hate repeating themselves. Hackers usually work with a puzzle-solving approach. The puzzle is considered as finished as soon as it has been solved. Solving the puzzle a second time is stupid because it has already been solved. I asked myself: Couldn’t a new approach be to solve the puzzle over and over again? To repeat the puzzle in an endless loop, like a bodybuilder in the gym works through his training program over and over again and builds up physical wealth, while hackers almost never build up physical wealth and become weaker and weaker under the increasing weights of technological progress? Maybe I’m a bit crazy, but with the staggering progress of large language models, Devin AI, and inference with increasing levels of education, I like to think that a Shaolin approach could countermeasure weak codebases. That is: Write the same piece of code a million times. I suggest an approach that I stole from the gym. The approach is called Repetition and goes like this: Write code.
Delete the code written in Step 1.
Repeat. We learn through repetition. The same principle applied in machine- and deep learning: the learning curve moves towards the optimum by Gradient Descent, in literature often represented by a ball rolling into a valley. The ball starts on a slope and will roll down the slope due to gravity. At the bottom of the valley, the ball has reached its optimum. It converged on the best possible configuration of its available resources. When we put our intentions into action, we create momentum. If we keep creating momentum in a repeated manner, our cognition will gradually converge to the best possible configuration. Millions of repetitions later, and the novice has become a master. Because this “essay” is about hackers, I prefer to say: wizards with cognitive abs. This is not an attempt to force you to go to the gym; I’m just forcing you to remember the motivations that inspired you to choose excellence. Delete what you have achieved and achieve it again. And again, and again, and again. Maybe your bachelor’s thesis is still lying around somewhere? Rewrite it. Bill Gates hates this trick, but it might work. Bill Gates said in an interview that he looks for lazy engineers during the recruiting process at Microsoft. His statement really excited the lazy engineers. In programming classes, we repeated the D.R.Y. mantra hundreds of times - in computer science classes, as well as in professional literature. “Don’t repeat yourself,” they said. “Abstract, refactor, but keep your points of truth on a short leash,” they said, but I think there is a bit of delusion in this. Most coders out there have swallowed the blue pill, so they prefer to code the easy, 3rd-partied way. I see the cause of modern software degeneration to be exactly here. The IT industry has always been driven by competition. Netscape vs Microsoft, Playstation vs XBox, Windows vs Linux … Not to mention the many thousands of start-up entrepreneurs who try to win every year. And while global climate policy melts the ice of Antarctica into mush layer by layer, the skillset of contemporary software development is melting into mush as well - layer by layer. As if the competition eats itself just for the sake of staying competitive. Break Fast and Move Things The modern programming style treats libraries and APIs like first-class citizens. I sometimes find myself wondering whether what I am developing is still my own work or just a Lego technology meta-object cobbled together from 3rd party software. We computer people call it Abstraction. It means something like “standing on the shoulders of giants” and claims that it is wiser to build on existing technology rather than reinventing the wheel every time. This implies that the modern IT landscape stands on a foundation that it does not understand. This one Adonis guy in the outdoor pool with abs and a 1-to-n relationship between himself and the ladies accumulated his physical wealth through repetition. He climbed the train again and again because pains -> gains. I think we lost the connection to painful programming, and therefore, the stamina to finish groundbreaking endeavors. Regain it. Hackers are particularly good at one thing: Hacking. But hackers are particularly bad at another thing: Sports. Hackers who invest physical effort into their well-being are the minority. They use their bodies to perform organic duties, everything else falls into a clearly defined role model between brain and brain-container. The image of a pale, over-sugared and anti-social loner who invents Bitcoin at night in a disused factory in front of a fap station with 16 screens corresponds to the hacker’s ideal, but one thing usually becomes a problem with the hacker’s ideal: That the goal is to make the structure-porn that circulates in the hacker’s brain a reality, but structure-porn rarely succeeds on the first attempt. For structure-porn to get delivered successfully, many manual repetitions are necessary, and hackers hate repeating themselves. That the goal is to make the structure-porn that circulates in the hacker’s brain a reality, but structure-porn rarely succeeds on the first attempt. For structure-porn to get delivered successfully, many manual repetitions are necessary, and hackers hate repeating themselves. Hackers usually work with a puzzle-solving approach. The puzzle is considered as finished as soon as it has been solved. Solving the puzzle a second time is stupid because it has already been solved. I asked myself: Couldn’t a new approach be to solve the puzzle over and over again? To repeat the puzzle in an endless loop, like a bodybuilder in the gym works through his training program over and over again and builds up physical wealth, while hackers almost never build up physical wealth and become weaker and weaker under the increasing weights of technological progress? Maybe I’m a bit crazy, but with the staggering progress of large language models, Devin AI, and inference with increasing levels of education, I like to think that a Shaolin approach could countermeasure weak codebases. That is: Write the same piece of code a million times. I suggest an approach that I stole from the gym. The approach is called Repetition and goes like this: Repetition Write code. Delete the code written in Step 1. Repeat. Write code. Delete the code written in Step 1. Repeat. We learn through repetition. The same principle applied in machine- and deep learning: the learning curve moves towards the optimum by Gradient Descent , in literature often represented by a ball rolling into a valley. The ball starts on a slope and will roll down the slope due to gravity. At the bottom of the valley, the ball has reached its optimum. It converged on the best possible configuration of its available resources. Gradient Descent When we put our intentions into action, we create momentum. If we keep creating momentum in a repeated manner, our cognition will gradually converge to the best possible configuration. Millions of repetitions later, and the novice has become a master. Because this “essay” is about hackers, I prefer to say: wizards with cognitive abs. This is not an attempt to force you to go to the gym; I’m just forcing you to remember the motivations that inspired you to choose excellence. Delete what you have achieved and achieve it again. And again, and again, and again. Maybe your bachelor’s thesis is still lying around somewhere? Rewrite it. Bill Gates hates this trick, but it might work. Bill Gates said in an interview that he looks for lazy engineers during the recruiting process at Microsoft. His statement really excited the lazy engineers. In programming classes, we repeated the D.R.Y. mantra hundreds of times - in computer science classes, as well as in professional literature. “Don’t repeat yourself,” they said. “Abstract, refactor, but keep your points of truth on a short leash,” they said, but I think there is a bit of delusion in this. Most coders out there have swallowed the blue pill, so they prefer to code the easy, 3rd-partied way. I see the cause of modern software degeneration to be exactly here. The IT industry has always been driven by competition. Netscape vs Microsoft, Playstation vs XBox, Windows vs Linux … Not to mention the many thousands of start-up entrepreneurs who try to win every year. And while global climate policy melts the ice of Antarctica into mush layer by layer, the skillset of contemporary software development is melting into mush as well - layer by layer. As if the competition eats itself just for the sake of staying competitive. Break Fast and Move Things The modern programming style treats libraries and APIs like first-class citizens. I sometimes find myself wondering whether what I am developing is still my own work or just a Lego technology meta-object cobbled together from 3rd party software. We computer people call it Abstraction . It means something like “standing on the shoulders of giants” and claims that it is wiser to build on existing technology rather than reinventing the wheel every time. This implies that the modern IT landscape stands on a foundation that it does not understand. Abstraction This one Adonis guy in the outdoor pool with abs and a 1-to-n relationship between himself and the ladies accumulated his physical wealth through repetition. He climbed the train again and again because pains -> gains. I think we lost the connection to painful programming, and therefore, the stamina to finish groundbreaking endeavors. Regain it.

Attention Hackers - Work on Yourself

The Frame Defines the Picture I recently encountered a fascinating graphic puzzle. Trying to solve it brought to mind a concept frequently discussed in problem-solving and critical thinking. The puzzle is shown below. What is your answer? Problem-solving methods span a spectrum from unstructured to highly structured. Some individuals dive straight into the problem, while others may devise a detailed plan of action. There are also those who adopt a hybrid approach. Unstructured problem-solvers would’ve delved in and then realized — we need to understand what a “date” means. Structured problem-solvers would’ve probably questioned the term “date” first, and then decided upon the solution. Whichever route you took, you would at some time, and in some form, question the concept of “date.” You would’ve questioned the essential term or definition. In doing that, you would’ve automatically questioned the assumptions and concepts associated with the term. You would also have realized that your solution depends significantly on how you interpret the concept of a “date.” “There is no single right answer or path forward, but there is one right way to frame the problem.” — Clayton Christensen (Harvard Professor) What is a “date”? According to the dictionary, a date is: a social or romantic appointment or engagement. Clarifying the key term establishes the essential foundation for addressing the problem. Based on our context, we can further adapt this theoretical definition to a more practical version. The concept of a “date” may be interpreted as more romantic than social, or the other way around. Once the foundation is established, you can delve into the specifics with that knowledge as your guide. Examine the graphics — the handbags, suitors, settings, menu cards, etc. — to uncover additional clues and assumptions. Iterating this knowledge through various scenarios will lead to a solution. Could you reach a solution without defining “date”? Potentially. Such a solution, however, will not be cohesive or defensible. You will struggle to demonstrate the logic chain that underpins your answer. Plus, anyone else will also be able to convince you with their answer; an equivocal solution is not really a solution. This puzzle demonstrates that grasping key terms and definitions lays a strong groundwork for problem-solving. It’s akin to a concealed superpower. By discerning the crucial definition, you reveal underlying assumptions. These two influence how you frame the problem. How we frame the problem directly impacts the solution. Definitions + Assumptions = FrameFrame → Solution. So, next time you’re faced with a problem, pause for a moment. Understand the key definition(s). Think critically. Determine fundamentals. Unearth assumptions. Only then, attempt a solution. This may appear counterintuitive compared to our instinct to delve in, but it produces more cohesive solutions and is quicker than one anticipates. If you enjoyed this post, you will like my book 📗 Smart Problem Solving which provides practical guidance on problem-solving. The Frame Defines the Picture I recently encountered a fascinating graphic puzzle. Trying to solve it brought to mind a concept frequently discussed in problem-solving and critical thinking. The puzzle is shown below. What is your answer? Problem-solving methods span a spectrum from unstructured to highly structured. Some individuals dive straight into the problem, while others may devise a detailed plan of action. There are also those who adopt a hybrid approach. Unstructured problem-solvers would’ve delved in and then realized — we need to understand what a “date” means. Structured problem-solvers would’ve probably questioned the term “date” first, and then decided upon the solution. Whichever route you took, you would at some time, and in some form, question the concept of “date.” Whichever route you took, you would at some time, and in some form, question the concept of “date.” You would’ve questioned the essential term or definition. essential In doing that, you would’ve automatically questioned the assumptions and concepts associated with the term. You would also have realized that your solution depends significantly on how you interpret the concept of a “date.” “There is no single right answer or path forward, but there is one right way to frame the problem.” — Clayton Christensen (Harvard Professor) “There is no single right answer or path forward, but there is one right way to frame the problem.” — Clayton Christensen (Harvard Professor) Clayton Christensen Clayton Christensen What is a “date”? According to the dictionary, a date is: a social or romantic appointment or engagement. What is a “date”? According to the dictionary, a date is: a social or romantic appointment or engagement. Clarifying the key term establishes the essential foundation for addressing the problem. Clarifying the key term establishes the essential foundation for addressing the problem. Based on our context, we can further adapt this theoretical definition to a more practical version. The concept of a “date” may be interpreted as more romantic than social, or the other way around. Once the foundation is established, you can delve into the specifics with that knowledge as your guide. Examine the graphics — the handbags, suitors, settings, menu cards, etc. — to uncover additional clues and assumptions. Iterating this knowledge through various scenarios will lead to a solution. Could you reach a solution without defining “date”? Potentially. Such a solution, however, will not be cohesive or defensible. You will struggle to demonstrate the logic chain that underpins your answer. Plus, anyone else will also be able to convince you with their answer; an equivocal solution is not really a solution. This puzzle demonstrates that grasping key terms and definitions lays a strong groundwork for problem-solving. It’s akin to a concealed superpower. By discerning the crucial definition, you reveal underlying assumptions. These two influence how you frame the problem. How we frame the problem directly impacts the solution. Definitions + Assumptions = FrameFrame → Solution. Definitions + Assumptions = Frame Frame → Solution. So, next time you’re faced with a problem, pause for a moment. Understand the key definition(s). Think critically. Determine fundamentals. Unearth assumptions. Only then, attempt a solution. This may appear counterintuitive compared to our instinct to delve in, but it produces more cohesive solutions and is quicker than one anticipates. If you enjoyed this post, you will like my book 📗 Smart Problem Solving which provides practical guidance on problem-solving. If you enjoyed this post, you will like my book 📗 Smart Problem Solving which provides practical guidance on problem-solving. Smart Problem Solving Smart Problem Solving Smart Problem Solving

Pay Special Attention to Definitions When Solving a Problem

JavaScript is currently one of the most popular programming languages, according to many platforms. However, does popularity equate to being the most advanced or beloved language? It lacks certain constructs that are considered integral parts of other languages, such as an extensive standard library, immutability, and macros. But there is one detail that, in my opinion, doesn't receive enough attention - generators. In this article, I want to explain possible use cases for iterators and generators and how they improve the verbosity of one's code. I hope that after reading this article, the following piece of code will make all sense: while (true) {
    const data = yield getNextChunk();
    const processed = processData(data);
    try {
        yield sendProcessedData(processed);
        showOkResult();
    } catch (err) {
        showError();
    }
} This is the first part of a series: Iterators and Generators. Iterators So, an iterator is an interface that provides sequential access to data. As you can see, the definition doesn't mention anything about data structures or memory. Indeed, a sequence of null values can be represented as an iterator without occupying any memory space. Let's have a few examples: Array is probably the first thing that comes to mind when you think about iterators. It's a data structure that stores a sequence of values in memory. It's also an iterator because it provides sequential access to its elements. const arr = [1, 2, 3];
for (const item of arr) {
    console.log(item);
} The same goes for strings. They are stored in memory as a sequence of characters and provide sequential access to them. const str = 'abc';
for (const char of str) {
    console.log(char);
} Let's look at the following function example: const fn = () => Math.random(); This function can be considered an iterator because it provides sequential access to random numbers. Okay, why do we need iterators at all if arrays, one of the language's basic data structures, allow us to work with data both sequentially and in arbitrary order? Let's imagine that we need an iterator that implements a sequence of natural numbers, Fibonacci numbers, or any other infinite sequence. It would be difficult to store an infinite sequence in an array. We would need a mechanism for gradually populating the array with data and removing old data to prevent filling up the entire memory of the process. This unnecessary complexity adds extra implementation and maintenance overhead, whereas a solution without an array can be achieved in just a few lines of code: const getNaturalRow = () => {
    let current = 0;
    return () => ++current;
}; Iterators can also be used to represent data retrieved from an external channel, such as a WebSocket. In JavaScript, any object that has a next() method, which returns a structure with value (the current iterator value) and done (a flag indicating the end of the sequence), is considered an iterator. This convention is described in the ECMAScript language standard. Such an object implements the Iterator interface. Let's rewrite the previous example in this format: const getNaturalRow = () => {
    let current = 0;
    return {
        next: () => ({ value: ++current, done: false }),
    };
}; In JavaScript, there is also the Iterable interface. It represents an object that has an @@iterator method (accessible via Symbol.iterator constant) that returns an iterator. Objects that implement this interface can be iterated using the for..of loop. Let's rewrite our example once again, this time as an Iterable implementation: const naturalRowIterator = {
    [Symbol.iterator]: () => ({
        _current: 0,
        next() { return {
            value: ++this._current,
            done: this._current > 3,
       }},
   }),
}

for (num of naturalRowIterator) {
    console.log(num);
}
// output: 1, 2, 3 As you can see, we had to make flag "done" to change at some point, otherwise the loop would be infinite. Generators The next stage in the evolution of iterators was the introduction of generators. They provide syntactic sugar that allows values of an iterator to be returned as if they were the result of a function. A generator is a function declared with an asterisk function* and returns an iterator. The iterator itself is not explicitly returned; instead, the function yields values of the iterator using the yield keyword. When the function completes its execution, the iterator is considered done (subsequent next method calls will return { done: true, value: undefined }. function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
}

for (num of naturalRowGenerator()) {
    console.log(num);
}
// output: 1, 2, 3 Even in this simple example, the main nuance of generators is apparent: the code inside a generator function does not execute synchronously. The execution of a generator code occurs incrementally as a result of next method calls on the corresponding iterator. Let's examine how the generator code executes using the previous example. We will use a special cursor to mark where the execution of the generator is paused. At the moment of calling naturalRowGenerator, an iterator is created. function* naturalRowGenerator() {
    ▷let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
} Next, when we call the next method three times, or in our case, iterate through the loop three times, the cursor is positioned after the yield statement. function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current; ▷
        current++;
    }
} For all subsequent next calls, as well as after exiting the loop, the generator completes its execution. The result of subsequent next calls will be { value: undefined, done: true }. Passing parameters to iterators Let's imagine that we need to add the functionality to reset the current counter and start counting from the beginning in our iterator of natural numbers. naturalRowIterator.next() // 1
naturalRowIterator.next() // 2
naturalRowIterator.next(true) // 1
naturalRowIterator.next() // 2 It's clear, how to handle such a parameter in a custom iterator, but what about generators? It turns out that generators support parameter passing! function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
} The passed parameter becomes available as the result of the yield operator. Let's try to clarify this using the cursor approach. At the moment of creating the iterator, nothing has changed. Now, let's stop after the first next method call: function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = ▷yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
} The cursor is positioned after returning from the yield operator. With the next next call, the value passed into the function will set the value of the reset variable. But what happens to the value passed in the very first next call? It goes nowhere! If you need to pass an initial value to the generator, you can do so through the generator's arguments. Here's an example: function* naturalRowGenerator(start = 1) {
    let current = start;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = start;
        } else {
          current++;
        }
    }
}

const iterator = naturalRowGenerator(10);
iterator.next() // 10
iterator.next() // 11
iterator.next(true) // 10 Conclusion We have explored the concept of iterators and their implementation in JavaScript. Additionally, we have learned about generators, a syntactic construct for conveniently implementing iterators. Although I provided examples in this article with numeric sequences, iterators in JavaScript can solve a wide range of tasks. They can represent any data sequence and even many finite-state machines. In the next article, I would like to discuss how generators can be used to build asynchronous processes (coroutines, goroutines, CSP, etc.). JavaScript is currently one of the most popular programming languages, according to many platforms. However, does popularity equate to being the most advanced or beloved language? It lacks certain constructs that are considered integral parts of other languages, such as an extensive standard library, immutability, and macros. But there is one detail that, in my opinion, doesn't receive enough attention - generators. In this article, I want to explain possible use cases for iterators and generators and how they improve the verbosity of one's code. I hope that after reading this article, the following piece of code will make all sense: while (true) {
    const data = yield getNextChunk();
    const processed = processData(data);
    try {
        yield sendProcessedData(processed);
        showOkResult();
    } catch (err) {
        showError();
    }
} while (true) {
    const data = yield getNextChunk();
    const processed = processData(data);
    try {
        yield sendProcessedData(processed);
        showOkResult();
    } catch (err) {
        showError();
    }
} This is the first part of a series: Iterators and Generators. Iterators So, an iterator is an interface that provides sequential access to data. As you can see, the definition doesn't mention anything about data structures or memory. Indeed, a sequence of null values can be represented as an iterator without occupying any memory space. Let's have a few examples: Array is probably the first thing that comes to mind when you think about iterators. It's a data structure that stores a sequence of values in memory. It's also an iterator because it provides sequential access to its elements. const arr = [1, 2, 3];
for (const item of arr) {
    console.log(item);
} const arr = [1, 2, 3];
for (const item of arr) {
    console.log(item);
} The same goes for strings. They are stored in memory as a sequence of characters and provide sequential access to them. const str = 'abc';
for (const char of str) {
    console.log(char);
} const str = 'abc';
for (const char of str) {
    console.log(char);
} Let's look at the following function example: const fn = () => Math.random(); const fn = () => Math.random(); This function can be considered an iterator because it provides sequential access to random numbers. Okay, why do we need iterators at all if arrays, one of the language's basic data structures, allow us to work with data both sequentially and in arbitrary order? Let's imagine that we need an iterator that implements a sequence of natural numbers, Fibonacci numbers, or any other infinite sequence. It would be difficult to store an infinite sequence in an array. We would need a mechanism for gradually populating the array with data and removing old data to prevent filling up the entire memory of the process. This unnecessary complexity adds extra implementation and maintenance overhead, whereas a solution without an array can be achieved in just a few lines of code: const getNaturalRow = () => {
    let current = 0;
    return () => ++current;
}; const getNaturalRow = () => {
    let current = 0;
    return () => ++current;
}; Iterators can also be used to represent data retrieved from an external channel, such as a WebSocket. In JavaScript, any object that has a next() method, which returns a structure with value (the current iterator value) and done (a flag indicating the end of the sequence), is considered an iterator. This convention is described in the ECMAScript language standard . Such an object implements the Iterator interface. Let's rewrite the previous example in this format: ECMAScript language standard const getNaturalRow = () => {
    let current = 0;
    return {
        next: () => ({ value: ++current, done: false }),
    };
}; const getNaturalRow = () => {
    let current = 0;
    return {
        next: () => ({ value: ++current, done: false }),
    };
}; In JavaScript, there is also the Iterable interface. It represents an object that has an @@iterator method (accessible via Symbol.iterator constant) that returns an iterator. Objects that implement this interface can be iterated using the for..of loop. Let's rewrite our example once again, this time as an Iterable implementation: Iterable @@iterator Symbol.iterator for..of const naturalRowIterator = {
    [Symbol.iterator]: () => ({
        _current: 0,
        next() { return {
            value: ++this._current,
            done: this._current > 3,
       }},
   }),
}

for (num of naturalRowIterator) {
    console.log(num);
}
// output: 1, 2, 3 const naturalRowIterator = {
    [Symbol.iterator]: () => ({
        _current: 0,
        next() { return {
            value: ++this._current,
            done: this._current > 3,
       }},
   }),
}

for (num of naturalRowIterator) {
    console.log(num);
}
// output: 1, 2, 3 As you can see, we had to make flag "done" to change at some point, otherwise the loop would be infinite. Generators The next stage in the evolution of iterators was the introduction of generators. They provide syntactic sugar that allows values of an iterator to be returned as if they were the result of a function. A generator is a function declared with an asterisk function* and returns an iterator. The iterator itself is not explicitly returned; instead, the function yields values of the iterator using the yield keyword. When the function completes its execution, the iterator is considered done (subsequent next method calls will return { done: true, value: undefined } . function* yield next { done: true, value: undefined } function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
}

for (num of naturalRowGenerator()) {
    console.log(num);
}
// output: 1, 2, 3 function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
}

for (num of naturalRowGenerator()) {
    console.log(num);
}
// output: 1, 2, 3 Even in this simple example, the main nuance of generators is apparent: the code inside a generator function does not execute synchronously . The execution of a generator code occurs incrementally as a result of next method calls on the corresponding iterator. Let's examine how the generator code executes using the previous example. We will use a special cursor to mark where the execution of the generator is paused. does not execute synchronously next At the moment of calling naturalRowGenerator, an iterator is created. function* naturalRowGenerator() {
    ▷let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
} function* naturalRowGenerator() {
    ▷let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
} Next, when we call the next method three times, or in our case, iterate through the loop three times, the cursor is positioned after the yield statement. next function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current; ▷
        current++;
    }
} function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current; ▷
        current++;
    }
} For all subsequent next calls, as well as after exiting the loop, the generator completes its execution. The result of subsequent next calls will be { value: undefined, done: true } . next next { value: undefined, done: true } Passing parameters to iterators Let's imagine that we need to add the functionality to reset the current counter and start counting from the beginning in our iterator of natural numbers. naturalRowIterator.next() // 1
naturalRowIterator.next() // 2
naturalRowIterator.next(true) // 1
naturalRowIterator.next() // 2 naturalRowIterator.next() // 1
naturalRowIterator.next() // 2
naturalRowIterator.next(true) // 1
naturalRowIterator.next() // 2 It's clear, how to handle such a parameter in a custom iterator, but what about generators? It turns out that generators support parameter passing! function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
} function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
} The passed parameter becomes available as the result of the yield operator. Let's try to clarify this using the cursor approach. At the moment of creating the iterator, nothing has changed. Now, let's stop after the first next method call: next function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = ▷yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
} function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = ▷yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
} The cursor is positioned after returning from the yield operator. With the next next call, the value passed into the function will set the value of the reset variable. But what happens to the value passed in the very first next call? It goes nowhere! If you need to pass an initial value to the generator, you can do so through the generator's arguments. Here's an example: next reset next function* naturalRowGenerator(start = 1) {
    let current = start;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = start;
        } else {
          current++;
        }
    }
}

const iterator = naturalRowGenerator(10);
iterator.next() // 10
iterator.next() // 11
iterator.next(true) // 10 function* naturalRowGenerator(start = 1) {
    let current = start;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = start;
        } else {
          current++;
        }
    }
}

const iterator = naturalRowGenerator(10);
iterator.next() // 10
iterator.next() // 11
iterator.next(true) // 10 Conclusion We have explored the concept of iterators and their implementation in JavaScript. Additionally, we have learned about generators, a syntactic construct for conveniently implementing iterators. Although I provided examples in this article with numeric sequences, iterators in JavaScript can solve a wide range of tasks. They can represent any data sequence and even many finite-state machines. In the next article, I would like to discuss how generators can be used to build asynchronous processes (coroutines, goroutines, CSP, etc.).

Stories, News & Mentions about ATTENIO

Attenio’s Stories on HackerNoon

Attenio's News & Mentions