The Unlikelihood Baseline in Sentiment Experiments
by Writings, Papers and Blogs on Text ModelsAugust 26th, 2024
Too Long; Didn't Read
The unlikelihood baseline is a simple approach that maximizes the probability of the preferred response while minimizing the probability of the dispreferred response. However, it often produces meaningless results due to unconstrained likelihood minimization. The article includes examples of prompts used to evaluate summarization and dialogue performance.
Authors:
(1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier;
(2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier;
(3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier;
(4) Stefano Ermon, CZ Biohub;
(5) Christopher D. Manning, Stanford University;
(6) Chelsea Finn, Stanford University.
Table of Links
4 Direct Preference Optimization
7 Discussion, Acknowledgements, and References
A Mathematical Derivations
A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective
A.2 Deriving the DPO Objective Under the Bradley-Terry Model
A.3 Deriving the DPO Objective Under the Plackett-Luce Model
A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2
B DPO Implementation Details and Hyperparameters
C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details
C.2 GPT-4 prompts for computing summarization and dialogue win rates
D Additional Empirical Results
D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments
C.3 Unlikelihood baseline
This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.
L O A D I N G
. . . comments & more!
. . . comments & more!
