As AI models like ChatGPT get bigger, training them requires more and more computing power. Researchers are looking for ways to train these large without needing Google-scale resources. explores a new method called LongLoRA that can efficiently train models on much longer texts. AI models A new paper Why this matters Being able to train on longer texts allows the models to develop a deeper understanding and reasoning. This could let them answer questions that require more context, like summarizing a long research paper. The standard way of training these models on long texts takes a huge amount of computing power. For example, fine-tuning the 70B parameter LLaMA model on 32,000 tokens takes 128 high-end A100 GPUs! More efficient training means these powerful models can be created and adapted with more reasonable resources. This expands access beyond just the biggest tech companies. The core ideas The researchers focus on two main techniques: They use an attention pattern that looks locally instead of across the whole text. This "shift short attention" provides a good approximation during training while still allowing the full standard attention for inference. Approximating standard attention: Building on a technique called LoRA, they adjust only a small subset of weights rather than all of them. Plus, they allow some embedding and normalization layers to also be tuned. Improving low-rank adaptation: What they accomplished Using LongLoRA, they could fine-tune a 7B parameter model on texts up to 100,000 tokens long on a single 8-GPU machine. For comparison, previous work took 32 GPUs for only 8,000 tokens. The efficiency gains were huge - LongLoRA cut the training cost by over 10x for the larger context sizes. The models performed nearly as well as standard fine-tuning. For example, on a 32,768 token evaluation, the perplexity (a measure of prediction quality) was only 3% higher. Looking forward This shows the promise of more efficient training techniques to handle ever-larger models and contexts. There's still more progress needed to match the full quality of standard fine-tuning. But LongLoRA demonstrates that we can push towards training at a much greater scale without requiring unreasonable resources. More efficient training will help democratize access to powerful AI systems. Also published . here

This story contains new, firsthand information uncovered by the writer.

Researchers Decode Speech With Brain Waves: Can AI Read Minds Now?

Exploring Tool-Integrated Reasoning: Innovating Math-Proficient LLMs

Tell me why my site sucks so I can make it better. Be mean.

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Efficient AI Model Training: LongLoRA's Breakthrough in Handling Longer Texts

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Beginner's Guide to Understanding Unstructured Data Analysis with LangChain and DeepInfra

AI as the "Bad Student" in Class

AI Will Not Kill Quantum Computing

AI's Unstoppable Energy Appetite: A Looming Crisis

Beyond the Algorithm: How Training Data Can Make or Break a Generative AI Model

A Beginner's Guide to Understanding Unstructured Data Analysis with LangChain and DeepInfra

AI as the "Bad Student" in Class

AI Will Not Kill Quantum Computing

AI's Unstoppable Energy Appetite: A Looming Crisis

Beyond the Algorithm: How Training Data Can Make or Break a Generative AI Model

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps