Authors:
(1) Corby Rosset, Microsoft Research and Correspondence to corbyrosset@microsoft.com;
(2) Ching-An Cheng, Microsoft Research;
(3) Arindam Mitra, Microsoft Research;
(4) Michael Santacroce, Microsoft Research;
(5) Ahmed Awadallah, Microsoft Research and Correspondence to hassanam@microsoft.com;
(6) Tengyang Xie, Microsoft Research and Correspondence to tengyangxie@microsoft.com. Table of Links Abstract and 1 Introduction 2 PreliminariesDirect Nash Optimization 2.1 RLHF Based on Reward Models 2.2 RLHF with General Preferences 3 Direct Nash Optimization and 3.1 Derivation of Algorithm 1 3.2 Theoretical Analysis 4 Practical Algorithm – Iterative Contrastive Self-Improvement 5 Experiments and 5.1 Experimental Setup 5.2 Results and Analysis 6 Related Work 7 Conclusion and References Appendix A Extension to Regularized Preferences B Detailed Proofs C Additional Experimental Details C Additional Experimental Details Batched Prompting: We also show in Figure 3 the prompt that we send to GPT-4 to annotate preferences. For the sake of efficiency, we “batch” requests to GPT-4, meaning that instead of sending every pair of candidate responses to be annotated, we show all candidates side-by-side and ask GPT-4 to apply the scoring rubric to each one in the context window Cost Analysis: We also do a brief cost analysis associated with the scaled-up experiment on 600k training inputs. The major line items are the cost of sampling outputs, annotating them with GPT-4 to construct training pairs, and then training the next iteration against those pairs. For each of the six iterations: Sampling: it took about 18-24 hours to inference 5 outputs for all 100k examples on 10 8xA100 80GB pods, depending on the average length, costing about $6,000 based on spot pricing.


Annotation: the average number of prompt tokens sent to GPT-4 for annotation across iterations was about 450M, with an average of about 60M completion tokens, amounting to about $34,000 based on the version of the endpoint we were using.


Training: ironically, training was the cheapest step, taking only 12-24 hours on two 8xA100 80GB nodes Table 5: Outputs of the various DNO models across iterations for the question: When will the earth run out of fresh water?. The output for Iter-1 is a bit too long as shown in Table 2. We believe this could be addressed with better hyperparameter tuning or preference data. We fine the initial SFT model’s response to be missing important points about how the premise of the question is best addressed by highlighting access to freshwater. The last reponse, for Iter 3 is more informative and specific than the initial SFT response. Table 6: Outputs of the various DNO models across iterations for an interview question relating to the design of a URL lookup system.. The last response from Iter 3 is more informative and clearer, and doesn’t contain misleading information (searching a trie runs in linear complexity in the length of the strings, not constant...). The response for Iter 1 contains an implementation of a Trie, which may be unnecessary because the user did not ask for it. Table 7: Outputs of first and last DNO-More-Data iterations for the third example What factors led to the outbreak of ww1?. The last response from Iter 6 has a higher information density; it recalls more key facts and entities, including the specific date of the start of WW1, Triple Entente, the Triple Alliance, Gavrilo Princip, and the Black Hand. Iteration 1 also contains some of this information, but is too wordy. The initial SFT model seems a lot more superficial. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Corby Rosset, Microsoft Research and Correspondence to corbyrosset@microsoft.com; (2) Ching-An Cheng, Microsoft Research; (3) Arindam Mitra, Microsoft Research; (4) Michael Santacroce, Microsoft Research; (5) Ahmed Awadallah, Microsoft Research and Correspondence to hassanam@microsoft.com; (6) Tengyang Xie, Microsoft Research and Correspondence to tengyangxie@microsoft.com. Authors: Authors: (1) Corby Rosset, Microsoft Research and Correspondence to corbyrosset@microsoft.com; (2) Ching-An Cheng, Microsoft Research; (3) Arindam Mitra, Microsoft Research; (4) Michael Santacroce, Microsoft Research; (5) Ahmed Awadallah, Microsoft Research and Correspondence to hassanam@microsoft.com; (6) Tengyang Xie, Microsoft Research and Correspondence to tengyangxie@microsoft.com. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Preliminaries Direct Nash Optimization 2 Preliminaries 2.1 RLHF Based on Reward Models 2.1 RLHF Based on Reward Models 2.2 RLHF with General Preferences 2.2 RLHF with General Preferences 3 Direct Nash Optimization and 3.1 Derivation of Algorithm 1 3 Direct Nash Optimization and 3.1 Derivation of Algorithm 1 3.2 Theoretical Analysis 3.2 Theoretical Analysis 4 Practical Algorithm – Iterative Contrastive Self-Improvement 4 Practical Algorithm – Iterative Contrastive Self-Improvement 5 Experiments and 5.1 Experimental Setup 5 Experiments and 5.1 Experimental Setup 5.2 Results and Analysis 5.2 Results and Analysis 6 Related Work 6 Related Work 7 Conclusion and References 7 Conclusion and References Appendix Appendix A Extension to Regularized Preferences A Extension to Regularized Preferences B Detailed Proofs B Detailed Proofs C Additional Experimental Details C Additional Experimental Details C Additional Experimental Details Batched Prompting: We also show in Figure 3 the prompt that we send to GPT-4 to annotate preferences. For the sake of efficiency, we “batch” requests to GPT-4, meaning that instead of sending every pair of candidate responses to be annotated, we show all candidates side-by-side and ask GPT-4 to apply the scoring rubric to each one in the context window Batched Prompting: Cost Analysis: We also do a brief cost analysis associated with the scaled-up experiment on 600k training inputs. The major line items are the cost of sampling outputs, annotating them with GPT-4 to construct training pairs, and then training the next iteration against those pairs. For each of the six iterations: Cost Analysis: each Sampling: it took about 18-24 hours to inference 5 outputs for all 100k examples on 10 8xA100 80GB pods, depending on the average length, costing about $6,000 based on spot pricing. Annotation: the average number of prompt tokens sent to GPT-4 for annotation across iterations was about 450M, with an average of about 60M completion tokens, amounting to about $34,000 based on the version of the endpoint we were using. Training: ironically, training was the cheapest step, taking only 12-24 hours on two 8xA100 80GB nodes Sampling: it took about 18-24 hours to inference 5 outputs for all 100k examples on 10 8xA100 80GB pods, depending on the average length, costing about $6,000 based on spot pricing. Sampling: it took about 18-24 hours to inference 5 outputs for all 100k examples on 10 8xA100 80GB pods, depending on the average length, costing about $6,000 based on spot pricing. Annotation: the average number of prompt tokens sent to GPT-4 for annotation across iterations was about 450M, with an average of about 60M completion tokens, amounting to about $34,000 based on the version of the endpoint we were using. Annotation: the average number of prompt tokens sent to GPT-4 for annotation across iterations was about 450M, with an average of about 60M completion tokens, amounting to about $34,000 based on the version of the endpoint we were using. Training: ironically, training was the cheapest step, taking only 12-24 hours on two 8xA100 80GB nodes Training: ironically, training was the cheapest step, taking only 12-24 hours on two 8xA100 80GB nodes Table 5: Outputs of the various DNO models across iterations for the question: When will the earth run out of fresh water?. The output for Iter-1 is a bit too long as shown in Table 2. We believe this could be addressed with better hyperparameter tuning or preference data. We fine the initial SFT model’s response to be missing important points about how the premise of the question is best addressed by highlighting access to freshwater. The last reponse, for Iter 3 is more informative and specific than the initial SFT response. Table 6: Outputs of the various DNO models across iterations for an interview question relating to the design of a URL lookup system.. The last response from Iter 3 is more informative and clearer, and doesn’t contain misleading information (searching a trie runs in linear complexity in the length of the strings, not constant...). The response for Iter 1 contains an implementation of a Trie, which may be unnecessary because the user did not ask for it. Table 7: Outputs of first and last DNO-More-Data iterations for the third example What factors led to the outbreak of ww1?. The last response from Iter 6 has a higher information density; it recalls more key facts and entities, including the specific date of the start of WW1, Triple Entente, the Triple Alliance, Gavrilo Princip, and the Black Hand. Iteration 1 also contains some of this information, but is too wordy. The initial SFT model seems a lot more superficial. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Batched Prompting for Efficient GPT-4 Annotatio

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

The Art of Arguing With Yourself—And Why It’s Making AI Smarter

Direct Nash Optimization Beats Bigger Models with Better Data

The Best Way to Train AI: Reward Models vs Preference Optimization

How Direct Nash Optimization Improves AI Model Training

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

The Art of Arguing With Yourself—And Why It’s Making AI Smarter

Direct Nash Optimization Beats Bigger Models with Better Data

The Best Way to Train AI: Reward Models vs Preference Optimization

How Direct Nash Optimization Improves AI Model Training

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps