Unveiling Nuances: Multi-Token Prediction's Impact on Llama 2 Finetuning

Written by cosmological | Published 2025/07/22
Tech Story Tags: llama-2-finetuning | multi-token-prediction | llm-performance | coding-benchmarks | humaneval | mbpp | deep-learning | model-behavior

TLDR

Explore surprising finetuning results for Llama 2 with multi-token prediction (n=4) on coding benchmarks. Our analysis (Table S6) reveals that this loss, while promising, doesn't always yield significant improvements, possibly due to initialization challenges.via the TL;DR App

Table of Links

Abstract and 1. Introduction

3. Experiments on real data

4. Ablations on synthetic data

5. Why does it work? Some speculation

6. Related work

7. Conclusion, Impact statement, Environmental impact, Acknowledgements and References

A. Additional results on self-speculative decoding

B. Alternative architectures

C. Training speeds

E. Additional results on model scaling behavior

F. Details on CodeContests finetuning

G. Additional results on natural language benchmarks

H. Additional results on abstractive text summarization

I. Additional results on mathematical reasoning in natural language

J. Additional results on induction learning

K. Additional results on algorithmic reasoning

L. Additional intuitions on multi-token prediction

M. Training hyperparameters

D. Finetuning

Authors:

(1) Fabian Gloeckle, FAIR at Meta, CERMICS Ecole des Ponts ParisTech and Equal contribution;

(2) Badr Youbi Idrissi, FAIR at Meta, LISN Université Paris-Saclayand and Equal contribution;

(3) Baptiste Rozière, FAIR at Meta;

(4) David Lopez-Paz, FAIR at Meta and a last author;

(5) Gabriel Synnaeve, FAIR at Meta and a last author.

This paper is available on arxiv under CC BY 4.0 DEED license.

Written by cosmological | From Big Bang's singularity to galaxies' cosmic dance the universe unfolds its majestic tapestry of space and time.

Published by HackerNoon on 2025/07/22