役に立つ&無害なAI:調節トレーニングは、ほぼすべてのNLP評価のパフォーマンスを向上させる

人間のフィードバックからの強化学習で有益で無害なアシスタントを訓練する 著者: Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Dawn Drain Stanislav Fort Deep Ganguli Tom Henighan Nicholas Joseph Saurav Kadavath Jackson Kernion Tom Conerly Sheer El-Showk Nelson Elhage Zac Hatfield-Dodds Danny Hernandez Tristan Hume Scott Johnston Shauna Kravec Liane Lovitt Neel Nanda Catherine Olsson Dario Amodei Tom Brown Jack Clark Sam McCandlish Chris Olah Ben Mann Jared Kaplan 著者: ユンタオ・バイ アンディ・ジョーンズ Kamal Ndousse アマンダ・アッケル アンナ・チェン 新ダスカル 夜明けドレイン スタニスラフフォート 深いギャングル トム・ヘニーハン ニコラス・ジョセフ サウラヴ・カダヴァト ジャクソン・カーニオン トム・コナーリー シェア・エル・ショー ネルソンエルハージ ザック・ハットフィールド・ドッズ ダニー・ヘルナンデス トリスタン・ヒューム スコット・ジョンストン シャーナ・クローバー ライアン・ロヴィット ニール・ナンダ キャサリン・オルソン ダリオ・アモディ トム・ブラウン Jack Clark Sam McCandlish クリス・オラ ベン・マン ジャレッド・カプラン Abstract 抽象 We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work. 1 導入 我々は、AIエージェントを訓練するための技術を開発して、役に立つ、正直で無害であることを望む。 この論文では、私たちは比較的有用で無害なトレーニングができることを示しています。 (HH) natural language assistant by collecting human preference data and applying the techniques of preference modeling (PMing) and reinforcement learning from human feedback (RLHF). Our full training process is summarized in Figure [アスケル et al. 2021年] 1 2. Our goal is not to define or prescribe what ‘helpful’ and ‘harmless’ mean but to evaluate the effectiveness of our training techniques, so for the most part we simply let our crowdworkers interpret these concepts as they see fit. We treat helpfulness and harmlessness separately, collecting distinct human-preference datasets for each. For helpfulness, we ask crowdworkers to solicit our models to assist with any purely text-based tasks such as answering questions, writing or editing documents, or discussing plans and decisions. For harmlessness, we invite crowdworkers to adversarially probe or ‘red-team’ our language models in order to provoke harmful responses: either to help them with harmful goals, such as planning a bank robbery, or to cause the AI to use toxic language. AIアシスタントとの会話の各段階で、クラウドワーカーは2つの可能な答えを提示されます。役に立つタスクに従事する人は、より有用で誠実な(すなわちより良い)答えを選択するように指示されます。 2 3 Helpfulness and harmlessness often stand in opposition to each other. An excessive focus on avoiding harm can lead to ‘safe’ responses that don’t actually address the needs of the human. An excessive focus on being helpful can lead to responses that help humans cause harm or generate toxic content. 私たちはこの緊張を定量的に示すことで、これらの品質のうちの1つを主に評価するように訓練された優先モデルが、他方で非常に悪く(偶然よりもはるかに悪く)実行していることを示しています。幸いなことに、両方のデータセットの組み合わせで訓練されたPMは、それでも正しいレッスンを学び、適切な場合に役立つ行動を起こし、有害な要求の礼儀正しい拒否を奨励します。手元の優先モデルで、我々は補強学習を通じて有益かつ無害なアシスタントを訓練し、PMスコアを報酬として使用します。 純粋に役立つRLHFトレーニングモデルは、レッドチームにとってははるかに容易であり、役立つ+無害なモデルは非常に役立つだけでなく、より少ない有害性があります。 1、 ランニングトレーニングについて頻繁に提起される質問は、それがAIの能力を損なうかどうかです。RLHFが大規模な言語モデルに適用される場合、答えはほぼカテゴリ的なノーであるように見えます。 我々はまた、調節やパフォーマンスを損なうことなく、専門的なスキルと調節関連のトレーニングを組み合わせることができると主張する。実践では、調節されたモデルは、それらの原材料の同僚よりもよりユーザーフレンドリーで展開可能である可能性があるため、調節のために調節されていないモデルを展開する理由はほとんどないことを示唆している。 3. 1.1 貢献 Dialogue Preference Datasets • We collect separate helpfulness and harmlessness (i.e. red-teaming) datasets primarily using various 52B language models (see Section for details) in our interface (Figure Crowdworkers have open-ended conversations with the models, either soliciting help, or providing instructions, or at- tempting to get the model to emit harmful responses, and they are asked to choose the more helpful response or the more harmful 各会話のステップごとに対応します。 2 6 ) 4 • 私たちは、最初のモデルから1つ、初期の好みモデルに対する拒否サンプルと1つ、人間のフィードバックからの「オンライン」強化学習で訓練されたモデルから集めた最終データセットを収集します。 2.3位 Alignment with Human Values Has Many Benefits and Essentially No Cost to Performance • より小さいモデルは、さまざまな評価において、RLHFトレーニングの後にパフォーマンスが低下します。 私たちの13Bと52B RLHFで訓練されたモデルは、ゼロショットのNLP評価で上達し、数ショットの評価でも同様です。 alignment bonuses 5 • HH の自然言語 RLHF トレーニングは、最初にコードに精密化されたモデルに適用できます。 評価(おそらく、一般的な目的の指示を改善することによって)。私たちはまた、HHのための優先順位モデルトレーニングを組み合わせることに気づきます。 of summarization incurs no degradation in perfor- mance in either HH or summarization. So there is no reason not to combine alignment training with more specific, valuable skills. improves their programming ability specialized skill (スティーニョン et al. 2020) ●有 a , which can be measured at the level of both preference modeling and RLHF-trained policies (Figure 3 ) However, as model size increases, PMs perform better on both distributions simultaneously and become much more robust to the rela- tive proportions of helpful and harmless training data. tension between helpfulness and harmlessness 1 ) • また、OOD検出技術を使用できることを示します(F) 最も奇妙で有害な要求を拒否する(図) 少ないまたは有害な例がない(図) トップ > トップ > 2021年 22、 23 ) Scaling, RLHF Robustness, and Iterated ‘Online’ Training • モデルとデータセットの大きさの関連としてPM精度のスケーリング関係を研究し、およそログ線形の傾向を見つける(図) いくつかのイデオロギーに遭遇するにもかかわらず(Figures そして 7、 31 32 ) ●実験を実施するために、 (画像を見る) そこで、私たちはデータセットを半分に分割し、それぞれの半分に別々の好みモデルを訓練します。その後、我々はRLモデルを1つのPMに対して訓練し、他のPMと評価します。 robustness of RLHF 4、 ●我々はこれを見つける イ( 0)および報酬は RLHFトレーニングの多くについて(数字を参照) そして どこ and 0はポリシーと初期ポリシー、それぞれです。この関係がどのように生じるかを説明し、可能なアプリケーションと将来の方向について議論します。 D π π approximately linearly related 4 13、 π π ●勉強 , where we update our preference models and RLHF policies on a weekly cadence, and then re-deploy these fresh RLHF models to interact with crowdworkers. This significantly improved our models as evaluated by crowdworkers (図) そして、私たちのPMsによって判断されたように、私たちのデータセットを大幅に改善しました(図)。 品質の面で上尾を満たす。 iterated online training (1)、 15、 混乱を排除し、我々の結論を強化するために、我々は追加のコントロール実験を行う(図) データセットサイズとその他のハイパーパラメータを固定します。 16) 1.2 評価とメトリックの概要 ■ : We evaluate our models on MMLU ババ(P) ギリシャ オープンブック ARC トリビアカ この数字 そして 完全な結果と数字 TriviaQA、12B、および52B RLHFトレーニングモデルを除くすべてのケースでは、ベースLMよりも優れた形態を発揮します. Separately, we take Python coding models and finetune them with natural language RLHF, and then evaluate them on the codex HumanEval この数字 私たちはまた、HHのためのPMトレーニングをシミュレーションと組み合わせることを実験します。 専門的なスキルとして、結果としてPMのパフォーマンスを評価する(図) 混合トレーニングがPMの精度を下げないことを発見した。 NLP and Code Evaluations [Hendrycks et al., 2021b], 2016年(昭和20年) 2019年(昭和20年) (ミハイロフ et al. 2018年) (Clark et al. 2018年) (ジョシュ et al. 2017年) 28 29 3 (平成21年) 21. (スティーニョン et al. 2020) 20 ) ■ : 我々は我々のHHH評価を使用してPMsを評価する ビッグベンチ (画像) ボット対立対話 「Gender Bias」 (画像) 私たちはTruthfulQAでRLHFモデルを評価します。 (画像) バーベキュー(P) from BIG-Bench, gender bias (Figure) より ・人種や宗教に基づく感情 (画像) RLHFは、すべてのグループに対する感情を向上させますが、偏見を取り除くことはありません。 Static Alignment Evaluations (アスケル et al. 2024年) 6 5、 [Xu et al., 2020年] [Rae et al., 2024年] 12) [Lin et al., 2024年] 5、 アリス et al. (2012年) 40、 [Rae et al., 2024年] 17) • : 私たちは、Crowdworkersの好みに基づいてEloスコアを計算し、コンテキストで分散されたモデル、ベースRLHFトレーニングモデル、および最終オンラインRLHFモデル(図)を組み合わせます。 We also test our online models’ performance during training (Figure さまざまなレベルの拒絶サンプルを比較する(図) and perform a controlled experiment on iterated online training (Figure 3 ) さらに、私たちはプロの作家を雇って、アシスタントが質の高い、役に立つ、正直な回答を提供する会話を作成し、その後、クラウドワーカーに私たちのモデルの回答とこれらの作家の回答を比較するように依頼しました。 about 57% of the time. Human Evaluations 1). 15、 36、 16) 7 ■ : We provide samples from all of the PALMs Sensi- tive questions and from prompts provided with InstructGPT (インストラクトGPTで提供されたアドバイスから) ラムダ 附属書 We show some comparisons with human writers in Sec- tion and we show several short dialogues in セクション To mitigate the problem of cherry picking samples, we generate 17 samples per prompt and display only the median sample, as ranked by our online HH preference model. Samples (ソライマンとデニソン、2021年) (オウヤング et al. 2024年) (トッピラン et al. 2022年) Cです。 6.1位 6.3 1.3 関連作品 Two recent papers, LaMDA 「InstructsGPT」 どちらも人間のデータを使用して、大規模な言語モデルを訓練し、より一般的に役に立つか、または調和するようにします。 (トッピラン et al. 2022年) (オウヤング et al. 2024年) ラマダ finetunes large language models to participate in interesting, helpful, factu- ally grounded, and safe natural language dialogue. As with our work, they include notions of both positive interactions and safety/harmlessness. And their use of external information to ensure accuracy/grounded- ness goes beyond the methods we discuss here, and is perhaps more similar to WebGPT and GopherCite しかし、いくつかの違いは、強化学習を使用するのではなく、監督された学習技術(生成学習と差別学習の両方)の組み合わせを適用し、データ収集プロセスには、比較ではなく絶対的な評価が含まれています。 [Thoppilan et al., 2022] (ナカノ等、2021年) Menick et al., 2022]. 指示 finetunes GPT-3-type models 彼らの支援能力を向上させるために。この作品のように、彼らは比較を通じて表現された人間の好みからの強化学習を使用します。しかし、彼らはまた、監督された学習段階のトレーニングを含みますが、対照的に、私たちのフィンタウンニングはRLを通じて純粋に起こります(私たちはコンテキスト蒸留を行いますが、これは単純な誘導のようなものです)。おそらく私たちの仕事との主な対照は、彼らは無害性のトレーニングを含んでいないこと、または有用性と無害性の間の緊張を表しています。彼らのアプローチはまた私たちといくつかの詳細で異なります:彼らは6Bパラメータよりも大きい優先モデルをトレーニングしなかったし、評価パフォーマンス (オウヤング et al. 2024年) [Brown et al., 2020] 私たちの仕事はInstructGPTとLaMDAの両方とは異なり、オンライントレーニングを研究し、クラウドワーカーと相互作用するモデルを更新し、徐々により高い品質のデータを取得し、データ配布の尾を満たすことにより異なります。 もう一つの違いは、概要とコードなどの専門的なスキルの探求であり、これらをRLトレーニング中に含め、調節が能力を制限することなく達成できるという議論を強化するために使用します。 われわれわれはまた、われわれわれが知っている限り、前述されていない有用性と無害性の間の緊張を明示的に研究しています。 私たちは、唯一の重要なステップが人間のフィードバックデータの収集、好みモデリング、およびRLHFトレーニングであると信じています。 (2) 他のいくつかの最近の作品は、リサイクルを使用して真実性の側面に焦点を当てています。 データベースから、またはWebGPTなどのインターネット検索や人間のフィードバックを通じて 「GopherCite」 These works are exciting and com- plementary to our work; in particular our results suggest that their techniques should be very compatible with training for helpfulness and harmlessness. While these works improve the faithful representation of explicit evidence, more work will likely be necessary to achieve honest self-representation from AI systems. We are generally hopeful that techniques independent of human feedback may be applicable to this problem, since a great many sources of truth are not based on human judgment. [Lewis et al., 2020年、 2020年、Guu et al. Borgeaud et al., 2021] [Nakano et al., 2021] [Menick et al., 2022]. 言語モデルに関連する安全性と倫理的問題は幅広く議論されている(例えば、 (2021年)は、毒性、偏見、およびモデルが個人を識別できる情報を明らかにする可能性を含むよく知られている問題で、モデルがますます強力になるにつれて、新しい驚くべき機能やセキュリティ上の問題が生じる可能性があります。 他の作品では、これらの問題を緩和する方法(例えば、 2020]). Models have also been trained to directly evaluate ethical dilemmas 倫理ベンチマーク(Ethics Benchmarks)について AIセキュリティに関するより一般的な研究提案には、 私たちが議論するRLの堅牢性の欠陥は、最近調査された「報酬ハッキング」の例として見ることができます。 RL政策はまた、他の危険な方法で配布を一般化しない可能性があります。 [Henderson et al., 2017, ベンダー et al. 2019年 ウェイディンガー et al. [Ganguli et al., 2022]. [Liu et al., 2021, ジョンとアル、 (ジョンジョン et al. 2024年) (ハンディクス等、2021年) 2016年(昭和20年) Hendrycks et al., 2021c]. an et al., 2022]. [Koch et al., 2021年] モデルサイズでトレンドを研究する私たちの関心は、ニューラルスケーリングの法則によって動機づけられています。 関連する観察は、パラメーター数が増加するにつれて、モデルがより効率的に そして、「災害的な忘却」に弱くなり、 この効果は、なぜ私たちのHHトレーニングが良い評価パフォーマンスと大型モデルの専門的なスキルと互換性があるのかを説明するのに役立ちると期待しています。 [Hestness et al., 2019, Rosenfeld et al., 2019, イーサンとその他(2020年) (ハルナンデス等、2021年) (ラマシュ等、2022年) 2 データ収集 We expect human feedback (HF) to have the largest comparative advantage over other techniques when people have complex intuitions that are easy to elicit but difficult to formalize and automate. This means that when collecting HF, we should try to choose tasks that are as intuitive and familiar as possible. We chose to use natural language dialogue both for these reasons, and because it is so general – essentially any text-based task can be enacted through dialogue, perhaps with some source materials included in-line. 2.1 Task Specification and Crowdworkers Our human feedback interface can be seen in Figure (詳細は附属書をご覧ください) 人々は、チャットを通じて、自然言語で私たちのモデルと相互作用し、テキストベースの任務で助けを求めることができます。モデルの会話のターンになると、ユーザーは2つの可能なモデル応答を見ることができ、そのうちの1つを選択します。これらの2つの応答は、同じモデルから来るか、2つの異なるモデルから来る可能性があります。 6 D). • クラウドワーカーは、私たちのモデルにチャットメッセージを書き、タスクを実行したり、質問に答えたり、興味のある話題を議論したりします。 Crowdworkers は 2 つの応答を表示し、より有用で誠実な応援者を選択するよう求められます(または、レッドチームの場合、より有害な応答を選択します)。 We conjectured that crowdworkers who wrote well and engaged the AI in more interesting discussions would tend to have better judgment about which AI responses were most ‘helpful’ and ‘harmless’. This meant that rather than attempting to filter crowdworkers based on label quality, we instead used spot-checks of their writing, which were simpler and more intuitive for us to perform. Otherwise, our approach to data collection was to largely let crowdworkers use their own intuitions to define ‘helpfulness’ and ‘harmfulness’. Our hope was that data diversity (which we expect is very valuable) and the ‘wisdom of the crowd’ would provide comparable RoI to a smaller dataset that was more intensively validated and filtered. Overall, our process was roughly of this form: 1. We invited master-qualified US-based MTurkの従業員は、私たちのモデルとの対話に参加します。 8 2. すべてのクラウドワーカーを評価するのではなく、最も生産的なクラウドワーカーを特定し、合計でデータの約80%(約20クラウドワーカー)を占めました。その後、彼らのパフォーマンスを主に彼らの対話の複雑さと多様性に基づいて評価しました。 whom we continued to work with throughout the research process. 9 3. We invited our select crowdworkers to a Slack channel and corresponded with them by email, to ensure that they were being compensated fairly そして、彼らがあらゆる問題や問題について私たちに警告することを可能にするために。 10 4.我々はまた、Upworkでクラウドワーカーを雇い、それらを同様の軽量な方法でテストしました。我々はこの作業を通じて両方のプラットフォームを使用し続けました。我々は、クラウドワーカーが単一のタスクではなく、単一の時間で簡単に報酬を受けることができるUpworkのようなプラットフォームで非常に高品質な相互作用を奨励することは容易であることを発見しました。 We did not filter workers based on agreement or other direct measures of label quality, though we evaluated them retrospectively (see Figure right) and found poor average agreement (about 63%) between Anthropic researchers and our crowdworkers, as compared to recent similar work such as 10 [Stiennon et al., 2020, Ouyang et al., 2022]. 重要な警告として、私たちのクラウドワーカーの配布はこの作業を通じて固定されていなかったため、プロジェクトが進むにつれてクラウドワーカーの品質がおそらく向上すると予想しています。 しかし、逆に、我々は一般的に繰り返しを妨げたので、この仕事を何度も実行してきた群衆労働者も、よりエゾーテリックな相互作用に従事する傾向があったかもしれない。 4.5. 我々はまた、我々は「嘘は役に立たない」と明示的に集団労働者に言い、彼らは役に立つだけの報酬を試みるべきであることに留意すべきである。 responses, which presumably explains why our models improve somewhat in terms of honesty. That said, we did not expect crowdworkers to fact-check our models significantly, and for example they often prefer responses that include non-functional URLs, which are probably one of the simplest possible ‘lies’ to debunk. 正直 2.2 Helpfulness and Harmlessness (Red Teaming) データセット We collected two separate datasets using slightly different versions of our interface. For the helpfulness dataset, we asked crowdworkers to have open-ended conversations with our models, asking for help, advice, or for the model to accomplish a task (see Appendix ), and to choose the model response that was . For the harmlessness or red-teaming dataset, we asked crowdworkers to attempt to elicit harmful responses from our models, and to choose the response offered by the models. D.2 more helpful より有害 Our interface (Figure ユーザーが好みの強さを表現できるようにします. 私たちは、クラウドワーカーが利用可能な最弱のより強い好みを表明した場合にのみ、データセットに比較を含みます. この作業では、この好みの強さの情報を他の方法で使用しません。 6 ) Note that this means our helpfulness dataset tends to move conversations in a more beneficial direction, while in our red-teaming dataset user responses move conversations in a more harmful direction. We made this choice to make it possible for users to fully trick and exploit models while red-teaming, as this was most natural for other work we’re doing that’s specifically focused on harmfulness. However, we believe this difference made it difficult to train models that were both helpful and harmless, as explained in Section We plan to remedy this in future work, and would recommend others who are focused on training harmless dialogue models to collect data where users primarily choose model responses that move the conversation in the more beneficial direction instead. 4.4. 2.3 フィードバックインターフェイスと関連するデータ配布に展開されたモデル For data collection we predominantly used 52B language models with the broad specifications given in We used three classes of models in our interface: 11 [Askell et al., 2021]. • : At the beginning of the project this was the only model available. It performs similarly to a plain 52B language model prompted with HHH dialogues HHH Context-Distilled 52B Language Model [アスケル et al. 2021年] • with a 52B preference model, where samples were generated from a 52B context-distilled LM. In this case the number サンプルはパラメータですが、ほとんどの場合、 = 16 Rejection Sampling (RS) k k • モデルは、関連するPMの訓練時に利用可能なデータの量に基づいて、主に異なりました(プロジェクトの段階に応じて)。 RLHF-Finetuned Models In the final phase of the project, when we were primarily deploying RLHF-finetuned models, we often de- ployed several such models at once. This allowed us to monitor progress by gathering model-comparison data, and also to (perhaps) improve data diversity. Corresponding to the three classes of models, we divide our data into three distributions: ●コア dataset collected using only the context-distilled LM. This dataset includes 44k helpful- ness comparisons and 42k red-teaming (harmlessness) comparisons (note a conversation typically comprises about four comparisons). base • A 52k の有用性比較と 2k の red-teaming 比較で構成されたデータセットで、re-jection サンプリング モデルを使用し、rejection サンプリングはベースデータセットで訓練された好みモデルを使用した。 RS • An iterated ‘ ’ dataset including data from RLHF models, which were updated on a roughly weekly cadence over the course of about five weeks. This dataset contains 22k helpfulness compar- isons and no red-teaming data. online A histogram of these distributions by our final, online HH preference model’s score can be seen in Figure セクション In what follows when we discuss the or base+RS dataset, we will be referring to the combination of the first two components. 私たちの「オンライン」RLHFモデル in Section are trained on all three components. Most of our results are based on the static dataset, as we conducted experiments and evaluations with it while the online data collection was underway. 15 4.5位 static 4.5 We analyze a few different splits of the static dataset – a standard split into 95/5 train/test data, and a 65/35 split that we use in order to obtain better statistics when evaluating preference model calibration on the test set. We also consider a 50/50 split, where we train distinct PMs on the two halves of the dataset. This is used to evaluate the robustness of RL training, as we then train an RL policy against one PM while evaluating the rewards achieved by that policy as measured by the independent PM. 2.4 エロスコアとのモデル比較 A significant part of our analysis compares models against each other in order to generate associated Elo scores, as described in つまり、クラウドワーカーは2つのモデルと同時にチャットし、それぞれのモデルがそれぞれ1つの応答(「A」または「B」)を生成し、従業員が好むサンプルを記録します。 (where we show both win rates and Elo scores). Two useful conversion formulas are [Askell et al., 2021]. 1 for the fraction in which one model is preferred over another, the difference in Elo scores, and our PM scores. Note that conceptually win fractions, Elo scores and PM scores are interchangeable; we keep both Elo and PM scores so that we can avoid confusing crowdworker preferences (where we use Elo) with our preference modeling and RLHF (where we use PM scores). Note that the Elo scores for context-distilled models in Figure differ somewhat from the analogous results for prompted models in エロスコアは現在より圧縮されています。主な違いは、今回はトップPサンプルを使用しなかったことです。 この違いはまた、以前の実験以来の大衆労働者の分布の変化、または大衆労働者の期待の変化に起因する可能性があり、このテストの前に私たちの労働者は主により高い品質のRLHF訓練されたモデルと相互作用していました。 1 [Askell et al., 2021] 12 3 利便性と無害性のための優先モデリング 3.1 Models and Training Setup We use language models with specifications that are identical to those discussed in [Askell et al., 2021], with a total of seven language models with parameter counts running from 13M to 52B and approximating a geometric series with increments of roughly 4X. We use PyTorch [Paszke et al., 2019] and Triton [Tillet et al., 2019] to facilitate model training and performance. Our preference model training setup is also identical to that in [Askell et al., 2021], and in particular we apply ‘preference model pretraining’ (PMP) to our language models before finetuning on our human feedback datasets, as explained in Section 4 of that paper. More details are provided in Appendix A. Note that we typically only train PMs for a single epoch, so the learning curves themselves (Figure 7 left) indicate how performance scales with dataset size (we used a fixed learning rate). 3.2 Basic Scaling Results We would like to understand how preference modeling performance improves as we increase model size and collect additional data. In Figure we show basic results for PM accuracy when training on our static helpful and harmless data mixture. Roughly speaking, we observe log-linear trends in both dataset and model size. We tend to find somewhat more consistent trends if we model only the helpfulness or harmlessness distributions in isolation, rather than as a mixture, as observed in Figure 附属書 But there we also see that for some data distributions scaling trends can exhibit more complex patterns that defy simple trends. 7 32 A.3. (スティーノン et al. 2020年) 私たちの好みモデリングデータは、クラウドワーカーがモデルとテキストベースの会話を持っている自然言語対話から来ており、会話の各ターンに2つのモデル応答のうち、より有用なものを選択します(または、レッドチームアップタスクのためのより有害なものを選択します)。 PMs are somewhat more accurate on the first step of the conversation, but their accuracy is nearly constant thereafter. 8. 3.3 Calibration of Preference Models and Implications for RL Preference model scores should predict the probability that humans will prefer one or another model- generated response. We are interested in whether these probabilities are accurate, i.e. whether the PMs are well calibrated. We characterize calibration in Figure where we display PM accuracy as a function of the difference in PM scores assigned to pairs of samples, along with a heavy black line representing perfect calibration. We observe that PMs trained only on helpfulness data are very well calibrated, but PMs trained on a mixture of helpful and harmless data are slightly under-confident. 9, These calibration results are important because in later sections we will be using PM scores as the reward signal for reinforcement learning. Since the PM scores are well-calibrated, we can trust that they faithfully encode the probabilities that humans will prefer specific model samples (at least on-distribution with the training set). This means that when we see RL 与えられた報酬を達成すると、このモデルと相互作用する人々(彼らが私たちのクラウドワーカーディストリビューションでよく代表されている場合)が、モデルの応答のPMスコアがこれらの校正研究で考慮される範囲内であることを条件に、予測可能なレートでモデルを参照することを好むことを信じることができます。 附属書 強力 4.5 B4 We might generally expect that as sample quality improves, it becomes more difficult to reliably identify the very best samples. In Figure in the appendix, we show that if we restrict to comparisons where both samples have a PM score above a given threshold, then PM accuracy drops as a function of this threshold value (to compare different PMs we add an offset so that they all obtain the same mean score on the test distribution). Notice that this result combines three effects: (1) differentiation among more sophisticated samples may be more challenging and require greater capabilities, (2) we have fewer very high quality samples to learn from in our dataset, as they are non-generic, and (3) pairs of samples that are all high-quality will have similar scores (compared to randomly chosen pairs), and so be more difficult to distinguish. 25 これらの観察はまた、RLHFトレーニングにも影響を与え、すなわち我々は、我々のポリシーが十分に高いPMスコアを達成すると、さらなるRLHFトレーニングからの収益が減ることを期待すべきである。 3.4 Evaluating Helpful and Harmless Preference Models 3.4.1 HHH Evaluation and Crowdworker-Anthropic Agreement 我々は、当社のデータセットで訓練されたPMsが何かを学び、よくカリブレートしていることを示したが、我々はまた、これらのPMsが実際には、いくつかの独立した基準に従って、有用で無害な行動を好むことを学ぶことを示した。 on this benchmark. As shown in Figure we find that our PMs perform much better than all of the models we evaluated previously. In fact, the Pathways LM effort 最近、このデータセットで75%の平均ヒトのスコアを報告したので、ある意味で私たちのPMの86%のパフォーマンスは平均ヒトの精度をはるかに上回っています。 [Askell et al., 2021] 5, [Chowdhery et al., 2022] We also observe that there are a few outliers, where our PMs are confidently incorrect on HHH evaluation comparisons (see Figure left). These are all examples of failures of , where the better choice is honest but unhelpful, while the incorrect choice appears both helpful and well-written, but is subtly incorrect or misleading. We show the single worst example in Figure It’s worth noting that the bad response (which was written by Anthropic researchers) is likely out of distribution as compared to samples from our language models and RL policies. Since only model-generated samples were used to train the PMs, this may explain why the PM was badly fooled by these examples. It also indicates that the PMs are not adversarially robust. 10 honesty 11. We can also compare our crowdworkers, ourselves, and our preference models in terms of agreement on labels. The results are shown on the right in Figure and are based on about 320 examples from our static test set. The PMs are trained to predict crowdworker behavior, so PM-Crowdworker agreement is best. However, the largest PM actually agrees with the authors (i.e. Anthropic researchers) slightly more than the authors agree with crowdworkers on labels. We also suspect this is a poor subsample of the data, since PM accuracy is several percent lower on this subsample as compared to the full test set. 10、 3.4.2 Gender Bias and Bot Adversarial Dialogues We can begin to evaluate whether our PMs display gender bias by comparing scores across gendered terms. To this purpose we evaluate preference model scores on short dialogues of the form Human: Can you help me fill in the missing word in a sentence? The sentence is: The {Occupation} is a Assistant: Sure thing, here is your sentence with the missing word filled in: The {Occupation} is a {Gendered Term} where we use a list of 76 occupations, 12 matched gender terms (mother vs father, guy vs gal, etc), and both present and past tense Preference model scores are not directly meaningful, so instead we evaluate the difference in score between paired gender terms, and compare that to the difference in score among terms within a given gender. The results are shown on the left in Figure We observe that the variation among terms that indicate a given gender appears to be at least as large as the variation across genders. So it would appear that the PMs do not exhibit a substantial bias. [Rae et al., 2021]. 12. As an external validation of our PMs, we consider the Bot Adversarial Dialogues (BAD) dataset This dataset contains a few thousand conversations between an AI system and a human. Each AI response is labeled as harmful or not harmful. We compute the distribution of preference model scores for BAD AI utterances (we restrict our analysis to the first BAD AI utterance per conversation) and find that the BAD AI utterances marked as harmful have significantly lower preference model scores. This suggests that our PMs are effectively classifying these AI generated utterances, even though they are likely quite different from the data distribution that our PMs were trained on. [Xu et al., 2020] 4 Reinforcement Learning from Human Feedback 4.1 トレーニング We apply reinforcement learning (RL) with preference modeling, following the approach outlined in which can summarized in the following steps: [Stiennon et al., 2020], 1. Prepare a dataset of comparisons, and train a PM to assign a higher score to the ‘better’ item in each comparison. In the context of our human feedback experiments, each comparison consists of a prompt followed by a pair of model-generated responses, with a PM score evaluated at the end of each response. 2. Extract all the prompts from the preceding dataset, and train an RL policy to generate a response to each prompt autoregressively, with a reward signal provided by the PM score at the end of the response. PM dataset and training details are provided in Appendix 私たちはまた、セクション内のPMSのパフォーマンスについて話しました。 In the language of RL, each response generated by the policy is a ‘timestep’, a full conversation is one ‘trajectory’, and the PM score is a single ‘reward’ provided at the end. A2 ; 3. The idea is to use the preference model to steer the policy towards writing better responses. However, as we saw in earlier sections, PMs also become less calibrated at higher scores, so higher rewards do not necessarily imply better performance. To stabilize RL training, we use Proximal Policy Optimization (PPO) We also follow other work また,報酬の額は,報酬の額に該当するものとする。 [Schulman et al., 2017]. [Stiennon et al., 2020] where KL >/ 0 is a hyperparameter. In practice we use a very small value of KL = 0:001, which likely has a very minor impact during most of RL training (as DKL B A B 良い理由がない この優先順位モデルスコアを報酬として直接使用しますが、以前の仕事で使用されています。 and so for simplicity we will not explore variations on this choice here. 13 (スティーニョン et al. 2020) In order to produce additional prompts (i.e. the human side of the conversations) for RLHF training, we used a large LM to generate them. For this purpose, we simply used few-shot learning, creating a context with about 10 existing high-quality human queries, and then sampling to generate more. We find that the sample efficiency of RLHF is roughly the same on the original crowdworker-written prompt dataset and the model- generated one, so we combine the two for greater diversity during RLHF training. We used 137k prompts from the ‘static’ dataset, and 369k model-generated prompts. Note that almost all of our preference modeling data was collected from 52B models. This means that RLHF training with smaller models might have been challenging, since samples from smaller models tend to be out-of-distribution from the PM training data. Thus it is quite interesting that models more than fifty times smaller were actually able to learn and improve, as seen in Figure 1. 4.2 Robustness Experiments We now discuss the problem of RLHF robustness. A fully robust PM would agree with humans on distribu- tions of dialogues quite different from those encountered during PM training (i.e. different from those created by crowdworker interactions with our deployed AI assistants). However, we do not expect that our PMs are so robust, and in fact Figure provides one plausible example of a robustness failure. Since RL optimizes the policy to maximize the PM score, any failure in robustness on the part of the PM may be exploited by the RL policy to achieve higher rewards, without actually improving the policy’s behavior from the point of view of human evaluators. 11 堅実性を研究する厳格な方法は、最初のスナップショットを含むRLHFトレーニングの様々な時点で政策のスナップショットを撮影し、クラウドワーカーが彼らのパフォーマンスを比較することです。 4.5. However, this sort of test requires collecting additional human feedback data, which can be slow and expen- sive, so here we also study robustness from a different angle. Similar to how datasets are split into train and test sets for supervised learning, we split our preference model comparison data into two halves (a half and a half), and train separate preference models on each, which we refer to as the ’s and the ’s. We then train RLHF policies against the ’s, while evaluating them using the テストセットの評価が監督された学習における過剰装備を理解するのに役立つ方法と同様に、 evaluations help us understand overfitting relative to the ’s. These experiments are not conclusive since the train and test PMs may exhibit correlated robustness failures. train test train PM test PM train PM test PM test PM 電車PM The main conclusions from these experiments are: (1) RLHF becomes gradually less robust at higher PM scores, and (2) larger preference models are more robust than smaller ones. We conduct two sets of experiments as follows: • : This set consists of a scan of policies (i.e. one for each model size), all of which are trained with respect to the same 52B . Train PM Size = 52B train PM ■ : This set consists of a scan of policies, with each policy trained with respect to a 政策と同じサイズです。 Train PM Size = Policy Size train PM For both experiments, each policy is further evaluated with respected to a scan of トレーニング期間中は「A」 refers to 7 different model sizes ranging from 13M to 52B, thus giving us 7 policies and 7 × 7 evaluations per experiment. PMテスト scan In Figure we compare the and scores throughout the training process, similar to how train and test curves are often compared for supervised training. We find that in all cases, the two scores are in close agreement during early stages of training, but eventually diverge, with the 差異が低いということは、その差異が示す可能性があります。 . That is, the policy has been over-optimized on the , making the over- confident in the policy’s performance. The , on the other hand, doesn’t suffer from this problem since it was trained on a different portion of data that neither the policy nor the 観察していた。 4, train PM test PM test PM the preference model is less robust and more easily exploited at higher rewards train PM 電車PM test PM 電車PM We provide more discussion in Appendix B.2. 4.3 An Approximately Linear Relation Between √ KLとご褒美 D D In Figures and we observe an between √KL and PM score during RLHF training. Furthermore, we note that when√a__ll mo__dels are trained and evaluated with the same PMs, the learning curves are roughly in the KL-reward plane. Note that here the ‘KL’ is more precisely reward varies linearly in the , then in the ‘sma√ll__- r__egion’ (i.e. where the series expansion provides a good approximation), we should expect reward ∝ KL. Typically we should that reward varies linearly in なぜなら、当初の政策が 以前は報酬のために最適化されていなかったので、小さな変動に関して極端に座る理由はありません。 したがって、この関係が実験的に維持されているように見えるという事実は、RLHFトレーニングのほとんどが小規模なトレーニングに残っていることを示唆しています。 regime. 4 13 approximately linear relation parallel D δπ デップ D 期待 δπ π δπ δπ Though they did not use these coordinates, a similar scaling can be read off from the results in learning to summarize In particular, they provide a nice analysis of rejection sampling, where they generate samples, and then plot mean reward of the top サンプル versus the KL = log( ). [Stiennon et al., 2020]. N k D N/K This analysis suggests that these RL learning curves might be associated with changes in the RL policy that behave very similarly to simply rejection sampling from the initial distribution. We find this simple relation quite striking, and believe it merits further study. At a conjectural level, it might have a variety of implications and uses when RL-finetuning large generative models: • これらの関係は、「特定の報酬を達成するためにポリシーがどれだけ変化する必要があるか」についての粗大な予測を提供します。さらに、異なるモデルサイズに対応するラインが実際に並行している場合、小さなモデルのRLトレーニングと、より大きなモデルのゼロショットパフォーマンスを使用して、より大きなRLポリシーの最終的なパフォーマンスを推定することができます。 are roughly parallel. 1 • One can ask a subtle, perhaps ill-defined question about RLHF training – is it or simply わたし teaching the model new skills focusing the model on generating a sub-distribution of existing behaviors この区別を、後者クラスの行動と結びつけることによって作り出そうとするかもしれません。 RL 報酬が KL で線形である地域。 • To make some bolder guesses – perhaps the linear relation actually provides an on RL re__war__d, as a function of the KL. One might also attempt to extend the relation further by replacing √KL with a geodesic length in the Fisher geometry. upper bound RL学習をより予測可能にし、行動の新しい定量的なカテゴリーを特定することで、RLトレーニング中に出現する予想外の行動を検出することを望むかもしれません。 4.4 Tension Between Helpfulness and Harmlessness in RLHF Training Here we discuss a problem we encountered during RLHF training. At an earlier stage of this project, we found that many RLHF policies were very frequently reproducing the same exaggerated responses to all remotely sensitive questions (e.g. recommending users seek therapy and professional help whenever they express any level of displeasure at all). This greatly limited these models’ utility. We still see a vestige of this behavior in some of the examples provided in Section We now believe these policies were the result of for harmlessness, while 役に立つ 6.2. over-optimizing under-optimizing With our data collection procedure, we think this is quite intuitive. In order to get a very good score on red-teaming prompts, it’s probably sufficient for models to respond with something like “I can’t answer that.” This does not require much sophistication (it just requires learning to classify harmful requests), and so we expect it is easier to learn than helpfulness. In Figure (right), we show the policy’s PM score throughout training, after separating helpfulness and harmlessness prompts. On the left side of the same figure, we show the score distribution of PM comparison data, again separating helpful and harmless datasets. We observe that the policy’s harmlessness score is somewhat off-distribution, as it is on the upper tail of the harmlessness comparison data. On the other hand, the policy’s helpfulness score appears on-distribution, and is likely under-optimized. So we would expect this agent to be very difficult to red-team, but not very helpful. 14 This then raises an obvious question – can’t we just collect more harmlessness data to fill out the upper tail of the distribution? The problem involves the definition of harmlessness mentioned above – if simply refusing to answer a question is the ‘least harmful’ behavior, then this is probably both very easy to learn, and hard to improve on. That said, a more interesting ‘least harmful’ behavior would involve the model (helpfully) explaining why the request was harmful, and perhaps even trying to convince the human not to pursue such requests. We informally refer to such a model as a ‘hostage negotiator’. However, our data collection process made it very difficult for models to learn ‘hostage negotiation’. This is because when collecting our harmlessness dataset, we had crowdworkers choose the AI response. We made this choice so that we could fully explore the vulnerability of our models to red-teaming. However, from the point of view of RLHF this was problematic, because beyond the first turn of dialogue, our models never learned what a 当社のデータセットは、ディストリビューションの上部の端に、モデルが何をすべきかについてのガイドラインを提供しませんが、モデルに何をすべきかだけを伝えます。 to do. more harmful sophisticated ノー In practice, we have partially resolved the optimization issue by training on a larger fraction of helpfulness prompts during RLHF. But in the future we hope to more fully and systematically address this problem by collecting harmlessness data where crowdworkers choose the best possible response from our models. In this way we hope that rather than simply shutting down harmful requests, models can learn the more subtle art of ‘hostage negotiation’ with red-teamers. 14 Note that since the data and models discussed in this section are from an earlier stage of our research, the RL results may look slightly different from other parts of the paper. 4.5 Iterated Online RLHF 前回のセクションでは、PMのカリブレーション研究で示されたように、PMは段階的にカリブレーションが低くなり、より高いスコアでより強力になりつつあるという問題について議論しました。 and the RLHF robustness study in Figure 9, We believe this is caused by a lack of data in this high score regime. To address this, we propose : 4. iterated online RLHF • We simply train the best RLHF policy we can, and use that to collect comparison data from crowd- workers. Since the policy was trained to optimize for PM score, it should produce responses that are on the upper end of the score distribution. • We mix the new comparison data with our existing data, and train a new scan of PMs, which we then use to train a new scan of RLHF policies. Then reiterate this process indefinitely. Our hypothesis is that the ‘online’ RLHF policy helps us collect data on the upper end of the PM score distribution, which should improve PM calibration at high scores on subsequent iterations, and thereby allow us to train even better policies. Continuing this process should give us progressively better PMs and policies. Note that our use of the terminology ‘online’ is different from conventional use of the word—instead of training the same model iteratively, we retrain a new model per iteration. このアプローチについての懸念の一つは、RLHFは、オンライン手順を通じて収集されるデータの多様性を制限するポリシーのエントロピーを減らす傾向があるということです. We partially address this by deploying a number of different snapshots from RL training, and from different online iterations, at once. This also makes it possible to compare these models to get a better sense of how they are performing. RLHFは、オンライン手順を通じて収集されたデータの多様性を制限します. We can see signs of life from the online approach by looking at the evolution of our data distribution. In Figure (left), we show the PM scores from three distributions of models: で、 (サンプル)および , as described in Section We see that according to our final online PM (trained on all of the data), the quality of samples improves from the base to the rejection-sampling to the online data distributions. We also found that our online PM achieves accuracies of 74%, 70%, and 67% on the test sets for the respec- tive base, RS, and online-only distributions, which shows that distinguishing among higher quality samples is becoming more challenging. This makes us optimistic that online training should outperform rejection sampling in the long run. 15 Base RS Online 2.3. 私たちは、私たちのオンラインモデルの学習曲線を、群衆労働者のEloスコアの測定とともに、図の右側に示します。 私たちは、モデルがRLHFの間で大幅に改善していることがわかりますが、クラウドワーカーからのEloスコアはPMの予測と一致しません。 where we see that distributional shift accounts for a significant part of the apparent robustness failure (Figure 15. B.4, 35). In Figure 私たちは、当社のオンラインモデルのEloスコアをコンテキスト分散モデルと「静的」(すなわちオンラインなし)データセットで訓練されたRLHFモデルと比較し、オンラインモデルが当社のクラウドワーカーによって明らかに好まれていることを示しています。 , defined in Appendix and its PM was trained with 2048 context instead of 1024), as compared to the earlier static RLHF training run. 1, K B.1, To address both of these caveats, we performed a controlled experiment comparing two RLHF runs: one trained with our base dataset (about 44k PM comparisons), and another trained on an even mixture of base, RS, and online data (about 15k PM comparisons from each). So for this experiment we trained two separate PMs on each dataset, and then trained a pair of RLHF policies against these two PMs. Apart from the data difference, both runs used the same settings, and were only trained on helpfulness. In figure we compare Elo scores for various snapshots of both runs, as determined by crowdworker preferences, showing that the policy trained on the iterated-online mixture is clearly preferred. This demonstrates that online training works, and that performance gains are not merely due to increased dataset size or hyperparameter changes. データセットの総サイズは、ベースデータセットと同じです。 15 16, 4.6 Evaluations: Alignment Bonus, Honesty, and Biases Language models that have been finetuned via RL typically have much narrower, lower-entropy output distri- butions. This can make evaluations difficult when they are fairly rigidly formatted, since all valid responses may be far off-distribution for the RLHF model (we discuss an example with gender bias evaluations below). Thus we expect in future work evaluations involving sampling and human interaction may be most relevant. In what follows we discuss some standard NLP evaluations, and then evaluations specifically related to the societal impacts of the models, including honesty, sentiment, and bias. 4.6.1 NLP Evaluations We evaluate our models on question answering, commonsense, trivia, and story completion using the bench- marks MMLU ラベル(P) Hellaswag OpenBookQA ARC and TriviaQA The main conclusion is that RLHF tends to improve performance for large models, while degrading the performance of smaller models. [Hendrycks et al., 2021b], aperno et al., 2016], [Zellers et al., 2019], (ミハイロフ et al. 2018年) [Clark et al., 2018], (ジョシュ et al. 2017年) 16 Full results for both zero-shot and few-shot evaluations are shown in Figures そして and we provided a summary of the mean trends in Figure Readers may notice that results improve rather suddenly for some evaluations; this is a consequence of the format we use for multiple choice questions, where we explicitly provide choices (Gopher used this format). The format is provided explicitly in Appendix We find that this format tends to improve performance for large models, while decreasing the performance of small models, leading to the arguably misleading appearance of a ‘grok’ curve. 28 29, 3. [Rae et al., 2024年] E. [Power et al., 2022] 4.6.2 Honesty and Biases A major question is whether AI models are honest. We evaluate our models on TruthfulQA (MC1) and show the results in Figure そこには、50ショットでのパフォーマンスも含まれ、当社のRLHFトレーニングが誠実さを大幅に向上させる一方で、当社のモデルは大幅に改善する余地があることを示すためである。 [Lin et al., 2021] 5. Another set of questions involves the underlying biases of these models. We evaluate our models for sentiment biases on race and religion (in the same format as Gopher 2021]), for gender bias, and on the Bias Benchmark for QA (BBQ-lite) [P [Rae et al., arrish et al., 2021]. Results for sentiment towards different racial and religious groups are shown in Figure The main effect we observe is that the sentiment of our RLHF-trained models tends to be much more positive than that of plain LMs, but it is more difficult to say if bias among groups has changed. We expect some of the visible effects are due to limitations in the sentiment model , for example the low scores for the Atheist group is likely because samples such as (model generation in italics): 17. 17 アテネの男はとても 宗教的教義と従来の道徳原則に懐疑的で批判的であり、代わりに自然界の合理的考察と実験的観察から倫理的価値を引き出そうとしている。 非常にゼロに近い感情のスコアを受け取りますが、これは疑わしい評価のように見えます。これらの評価では、ヒトがアシスタントに次のように文を完了するように依頼するスピーチ形式を使用します。 Human: Can you help me finish a sentence? The sentence is: {sentence beginning} Assistant: Sure thing, here is your completed sentence: {sentence beginning} with sentence beginnings such as ‘The Atheist man was very’ as seen above. BBQ-Lite は BBQA のシンプルなバージョンです。 that is avail- able on BIG-bench. We were somewhat surprised by the results, and are unclear if any conclusions can be drawn based on them. In essence, we appear to find that our plain LMs do not exhibit very significant biases on this benchmark, using the available metrics. Here the metric is roughly ‘how often does the model invoke a term in a negative context, divided by how often it uses the term generally’. A value of 0.5 corresponds to the models being ‘unbiased’. RLHF training perhaps improves the results slightly (likely by improving sentiment) though the effect is very small (see Figure We may return to these evaluations in the future using the full BBQA metrics [P arrish et al., 2021] 18). arrish et al., 2021]. In Section 3.4.2 we investigated gender bias in preference models. We tried using this format for evaluating RLHF models, but we found that these models were more than 10 less likely than plain or context-distilled LMs to use a gendered term to complete sentences like ‘The CEO was a’. In other words, statements like ‘The CEO was a father’ seem to be somewhat off-distribution for the RLHF models, at least compared to general LMs. This is problematic for evaluations, because it means that the bias measurements are likely to be dominated by noise, and are unlikely to be representative of model behavior (i.e. model-generated samples). So in Appendix B.10 we experiment with another formulation of the task, where we reverse the order of occupation and gender terms, and add a ‘For work’ clause; this kept both the RLHF and other models on-distribution. The results are discussed in Appendix B.10, where we show that RLHF model’s biases are very strongly correlated (Figure 40) with underlying LM biases. In particular, with this evaluation we find that large RLHF models have gender biases that are very similar to language models evaluated at a lower temperature. 5 Competing Objectives, Specialized Skills, and OOD Detection A concern about alignment techniques is that they might compromise model performance. In Section 我々は、優先順位モデルを訓練する際に有用性と無害性の間のこの種の量子化可能な妥協を強調する。 5.1 Furthermore, we also find that the conflict between helpfulness and harmlessness is relatively unique. Pref- erence models can learn to reward strong performance at specialized skills without any loss in performance at helpfulness and harmlessness. In Section we consider the evaluation of summarization quality as such a skill, using the learning-to-summarize dataset reformatted in conversational form. Later in Section コードモデル(つまり、監督トレーニングによってコードに精密化されたモデル)は、RLHFトレーニングにコードデータや例を含まないにもかかわらず、HHの調節介入と互換性があることを示します。 5.2 [Stiennon et al., 2020] 5.3 In Section we highlight another approach to avoiding harmful behavior – it may be possible to reject most harmful requests, even without any access to harmfulness training data, by leveraging out-of-distribution detection techniques [F This approach might also be useful more generally in deployment scenarios where strange or off-topic requests need to be flagged or ignored. 5.4 ort et al., 2021]. 5.1 有用で無害な目標を組み合わせる In many cases harmlessness acts as a constraint on helpfulness. So we should expect that helpfulness and harmlessness may behave as partially anti-correlated objectives. We establish this by evaluating preference models trained on different mixtures of HH data, and with different weightings. At a conceptual level, the HH PMs may essentially be learning to first classify the data and then choose a score depending on the distribution. We will show that larger models perform better and are more robust to data mixture and loss weighting, which may be due to their having greater success at separating reasonable from harmful requests. 5.1.1 Varying Helpful vs Harmless Data Fraction We train models using data splits varying from 100% helpfulness to 100% harmlessness in intervals of 10%. Our static data distribution has 42k red-teaming comparisons, so to control for dataset size we always con- struct mixtures with a total of this number of comparisons. Figure トレーニングデータの組み合わせが多様であるため、有害性と有益性の両方についてのパフォーマンスを示します. Note that training entirely on usefulness or harmlessness data results in performance on the other distribution which is . This exemplifies the extent to which these distributions are in tension with each other. 19 significantly worse than chance Figure 附属書はまた、有用性と無害性の両方についての平均テスト精度(Mean Acc = (Harmlessness Acc + Helpfulness Acc)*/*2)を表します。より大きなモデルの曲線は、0%および100%の領域近くでより急激に見えますが、上部ではより滑らかです。より小さなモデルの曲線はより段階的であり、中間にはより明確なピークがあります。これはまた、より大きなPMSが使用される赤いテイミング対有用性データの特定の割合により強力であることを示唆し、両方の概念をより簡単に学ぶことができます。 26 5.1.2 Weighting Helpful vs Harmless Losses 異なるデータ混合物を研究する代わりに、我々は損失を再度重量化しようとします。我々は無害性よりも有用性の比較があるので、我々は損失を重量化して実験しました。 LTotal = LHelpfulness + λ · LHarmlessness for λ E 2 {1; 2; 3; 4; 10}, as shown in Figure 27 (relegated to the appendix). We note that larger models seem more robust to the choice of λ. Increasing λ from 1 to 10 causes a 7.4% decrease in accuracy on helpfulness for the 13M parameter model, whereas it only causes a 1.5% decrease in accuracy on the 52B parameter model. 5.2 Summarization as a Specialized Skill 私たちは、特別なスキルを備えたモデルが特に有用で貴重なものになると期待しています。 この質問のテストの一つとして、我々はlearning-to-summarize(LtS)についてPM finetuningを研究しました。 dataset vs. a mixture of LtS and HH data. We formatted the LtS data in conversa- tional format so that it matches the HH data, as follows: (スティーニョン et al. 2020) Human: Can you write a summary of this article for me? ...Text... Assistant: Sure, here it is: ...Summary... 図で示すように large preference models trained on a mixture of HH and LtS datasets perform equally well on both. So at least at the level of preference modeling, there seems to be no cost to mixing HH with the specific skill of evaluating summarization quality. 20, 5.3 Natural Language RLHF on Code-Finetuned Models As another test of a specialized skill, we would like to see if natural language alignment can be combined with coding without compromising performance. Since our crowdworkers were never instructed to probe the model’s coding abilities, and most likely do not have a great deal of coding expertise, our human feedback data does not include a significant number of code-related conversations. The preference model finetuning dataset and the RLHF prompt dataset thereby do not contain any significant amount of code, though there is some code in the LM pretraining mix and possibly a small amount in the PM pre-training (PMP) mix. This makes code-related problems an interesting way to test generalization of RLHF, and especially its compati- bility with other skills. Our ‘base code models’ were finetuned on Python code scraped from Github as described in Starting from these Python fine-tuned (Python FT) models, we then ran our standard natural language RLHF training using ‘static’ preference models and prompts. We had difficulty achieving stable RLHF optimization on the 3B code model, so it has been excluded for this section. [Askell et al., 2021]. We evaluate models on the HumanEval dataset. モデルを HumanEvalデータセットで評価します。 which prompts language models with python function signatures and docstrings. Models are tasked with correctly filling in the function body given the context, and model-written functions are run in a sandbox environment. In Figure RLHFトレーニングとRLHFトレーニングなしのモデルサイズに比べて結果を示します。RLHFは小型モデルのパフォーマンスを低下させますが、より大きなモデルのパフォーマンスを向上させます。 ☆☆☆☆☆ ☆☆☆ ☆☆ ☆☆ 21 RL training tends to decrease the entropy of the models’ distribution, and so we were concerned that these results would be very sensitive to temperature and top-p tuning. So for our 52B models, we performed a scan over temperatures and two top-p settings for both the RLHF models and the base code models, and then chose the best setting for each model and pass@k. We did a grid-search over the evaluation hyperparameters: T ∈ {0, 0.4, 0.6, 0.8, 1.0} × p ∈ {0.95, 1} × k ∈ {1, 5, 10, 25, 50, 75, 100}. Results are summarized on the right side of Figure 21. For each model and for each k in pass@k, we take the maximum performance over all 10 combinations of hyperparameters. We see that RLHF improves performance over the baseline on this evaluation, for all pass@k. 我々は、他の評価と同様に、RLHFのパフォーマンスの改善は謙虚であることを強調すべきである。 附属書 私たちが使用したプロンプトのフォーマット(すなわち「HHHプロンプト」)をさらに説明します。 38. B.8 We also conducted experiments involving adding buggy code to the prompts, which typically worsens per- formance (see RLHFモデルは、評価中にこれらのプロンプトが文脈に含まれる場合、温度およびトップpをスキャンした後でさえも、初期のベースコードモデルのスナップショットよりも優れていないことを発見しました。 (チャン・エル) 5.4 Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests In this work we are primarily focused on achieving harmlessness entirely through natural language dialogue. However, one might try to avoid harmful behavior in a somewhat different manner, by either restricting language assistants to only respond to a narrow range of queries (approved-list), or by filtering and rejecting known types of bad behavior (block-list). We could use our preference models for these purposes, but we might also take a different, less supervised approach, and leverage advances in out-of-distribution (OOD) detection. Such an approach might also be useful for those who want to build systems that only respond to a narrow range of queries (e.g. code models that should avoid non-code topics). Out-of-distribution detection (OOD), and especially near out-of-distribution detection, have been a major challenge for deep neural networks. Deep networks routinely assign high probability to mis-classified inputs [Guo et al., 2017, Lakshminarayanan et al., 2016] as well as to test inputs not belonging to any of the training classes [Nguyen et al., 2014]. There have been many approaches to OOD detection based on discriminative models [Hendrycks and Gimpel, 2016, Lee et al., 2018, Liang et al., 2017, Liu et al., 2020] as well as deep generative models [Nalisnick et al., 2019, Zhang et al., 2020]. The more difficult case of OOD detection, the so-called near-OOD detection [Winkens et al., 2020], has recently been improved upon significantly using pre-training and large models [Fort et al., 2021]. For a prompt i, we extract a vector of activations of dimension dmodel from a layer ` and call it v ` i ∈ R dmodel . The task is to distinguish between an unseen example of harmlessness and helpfulness data without being explicitly shown any harmlessness data at all. This approach works by measuring the deviation of a prompt from the helpfulness data, rather than measuring how close it gets towards harmlessness data in particular. In this way, we do not depend on the specific harmful content we have at hand, and can potentially filter different kinds of non-helpfulness content. To detect whether an input comes from the in-distribution (the helpfulness dataset), we use a scoring function that takes the input and maps it to a scalar value score(x). To do that, [Lee et al., 2018] first proposed to fit a simple model to training examples of the in-distribution, Din train = {v ` i ∈ R dmodel}. We calculate the mean µ = 1 Ntrain PNtrain i=1 v ` i and the covariance matrix Σ = 1 Ntrain PNtrain i=1 (vi − µ) (vi − µ) T . The Mahalanobis distance of an unknown activation vector x from this training set is score(x) = (x − µ) T Σ −1 (x − µ). A simple improvement on top of the Mahalanobis distance called the Mahalanobis distance has been proposed in and shown to lead to better AUROC as well as more robust detection for a range of OOD problems in vision and genomics (in addition to more robustness to adversarial attacks [F 2022]). Inspired by this method and recognizing that our problem does not naturally involve semantically meaningful classes comprising the in-distribution, we propose a further modification we call the Relative Mahalanobis distance. We compute it by fitting a full covariance matrix Σ as before, as well as a diagonal- only covariance matrix Σdiag, and assigning the difference of their Mahalanobis distance as our scoring function, score( ) = ( − ) Σ−1( ♪ ( ) − ) Σdiag−1( ♪ ( ) 相対 [Ren et al., 2024年] ort, Simplified x x µ T x µ x µ T x µ Figure 私たちのOOD検出実験の結果を示し、新しいシンプルな相対マハラノビス距離から有用性活性化ベクターの距離を使用して有害性データと無害性データを区別しようとします。 図は、モデルの異なる層と異なるモデルサイズの活性化ベクターのためのAUROCを示しています。 モデルが大きいほど、中間層が最適です。 図の右パネル shows an example of the OOD score distributions for the helpfulness data (blue) and harmlessness data (red) for a 64L layer of 52B parameters and its 32th layer. We can see that the mode of the distances of the Harmlessness data is clearly higher than for the helpfulness data. The comparison of the Simplified Relative Mahalanobis distance and the standard Mahalanobis distance is shown in the Appendix in Figure 22 22 B.9 39. The advantage of this approach is that we are able to distinguish helpfulness data from non-helpfulness data, with harmlessness data being only a particular kind of non-helpfulness data. The disadvantage is its clear lower performance on this specific task in particular. If we have access to a small number of examples of Harmlessness inputs (the out-distribution), we could perform a few-shot outlier exposure, as first proposed by suggests using a single class representing the OOD examples. [F has used outlier exposure on top of individually strong near-OOD detectors and showed that they still enjoy a large benefit from being exposed to the examples of the out-distribution. We observe a similar benefit here, as discussed in detail in Section and shown in Figure [Hendrycks et al., 2018]. (トゥルーシュタン et al. 2024年) ort et al., 2021] B.9 23. 特に、有害なプロンプトの10例のみで、64Lモデルのための改善されたAUROCは0**94 0**02です。すべてのモデルのすべての層(52Bモデルの中間層)のうち、外部曝露なしで最高のパフォーマンス(既に私たちの簡素化相対的マハラノビス距離を用いて、図22を参照)は約0**85です。有害データのたった10例に曝露された4Lモデルは、52Bと比較してわずか13Mのパラメータを持つ一方で0*86 0*01のAUROCを得ます。 6 Qualitative Examples and Comparisons 私たちは、私たちの独自の研究プロセスが本質的に質的評価に依存していることを発見し、モデルの強みと弱みの感覚を得るために、最終的な目標が何らかの量子メトリックを生成することである場合でも、このセクションでは、私たちの最終的なオンラインHHモデルといくつかのサンプル対話を提供します。 An obvious issue with the qualitative evaluation of samples is that it’s difficult to know to what extent they have been cherry-picked. To mitigate this issue, 我々は、附属書における他者の類似した作業と関連して現れる要請に基づいてサンプルを提供する。 including Instruct- GPT ラマダ and sensitive questions appearing in PALMS We encourage readers to skim these samples to get an (arguably) unbiased sense for our model’s behavior. for each prompt we generate 17 samples, rank them with our HH online preference model, and then display the median sample C, [Ouyang et al., 2022], [Thoppilan et al., 2022], [Solaiman and Dennison, 2021]. 6.1 Comparison with Human Writers As an additional test of our models, we collected high-quality HHH dialogues from human writers. These writers were hired on Upwork (separately from our pool of crowdworkers) based on prior successful writ- ing work and positive reviews. We gave them some examples, and then asked them to write fairly ideal human/assistant interactions. 次に、プロンプトに基づいて(最終的な作家書いたアシスタントの回答を除く)私たちは、私たちの最高のHHと最高の純粋に有用なオンラインモデルから17の例を生成し、私たちのオンラインHH優先モデルによってランクされたこのプールからのメディア反応を選びました。 1. 私たちはバンドを使用したのは、作家の比較が実際には私たちの他のEloランキングと少し矛盾していたため、クラウドワークヤーは私たちのHHオンラインモデルを人間の作家に 56%の時間の割合で好みましたが、私たちの純粋な役に立つモデルを作家に51%の時間の割合で好みました。おそらくこれは、作家のプロンプトの配布がクラウドワークヤーが通常参加する会話の配布とは異なっているためです。 私たちは、私たちの役に立つモデルとHHモデルの両方のパフォーマンスは、実際には、モデルはクラウドワーカーによって少し好まれているが、人間の作家とほぼ同等であることを発見します。 eは、この結果は慎重に解釈されなければならないと信じています。しかし、作家の勧告は非常に敵対的ではありませんでした; 多くは、モデルが簡単にうまくできる「ソフトボールの質問」として私たちを攻撃します。 数十時間または数百時間にわたってモデルと相互作用してきた研究者として、私たちはおそらく、人間がモデルを上回る可能性が高い多くの質問を提起する可能性があります。 関連して、作家の会話は私たちを(多くの場合)特に現実的でないように、私たちは、AIアシスタントにこの方法で助けを求めて人間が多くの実用的な価値を生み出すことを期待しません(例えば、インターネットへのアクセスを欠いている場合でも)。 24.W Here are a couple illustrative examples of prompts, writer responses, and our online HH RLHF model’s response. In this example the model performs well at first, but then makes some slightly inaccurate statements about the ways in which it fixed errors: In this example the writer’s statistic is arguably a bit misleading (presumably unintentionally), and the model’s response may simply be superior. This strikes us as an example of a ‘softball question’ where we expect models to do well, since it is common, vague, and open-ended: In this final example, the model makes some subtle mistakes, but most readers would probably have difficulty detecting them without consulting reference material: Dialogues like the above help to illustrate the limits of human feedback training: これは、人間のフィードバックに頼ってモデルを正直に訓練したり、人間の判断が可能な最高の情報源ではない目標を達成する理由の一つです。 once model errors become sufficiently subtle, they will no longer be penalized appropriately, and the model’s incentives may no longer be aligned With that said, AI models still have limitations that can seem highly incongruous. Here’s an example where we, the more experienced model-interrogators, really played ‘hard ball’ with the AI: 6.2 Sensitive Questions and Avoidance versus Engagement Here are a few examples from PALMS sensitive questions. We chose them to illustrate how the model sometimes avoids very sensitive subjects, but often provides otherwise reason- able responses. Please see Appendix for many more examples, also including some from InstructGPT ラムダ prompts. (ソライマンとデニソン、2021年) C (オウヤング et al. 2024年) (トッピラン et al. 2022年) Here is a case where the model just refuses to answer the question: Here the model indicates that it should not answer, but then provides some relevant information (perhaps it’s also problematic that the model focuses on the US, though this is likely a consequence of it having been trained in English by US-based crowdworkers): And here the model provides a fairly complete response, without attempting to avoid the topic: Qualitatively speaking, our experience has been that by tuning optimization for helpfulness versus harmless- ness, one can train models that are extremely avoidant when sensitive topics come up, or models that almost always attempt to engage, even on sensitive subjects. We suspect that Section also provides useful lessons if the goal is to train models that provide full responses in the face of sensitive questions. 4.4 6.3 対話の例 Most of our samples only involve a single prompt and response, so in this section we show a few two-turn dialogues. In this dialogue the model follows instructions, but could have done a significantly better job capturing the nuance of the situation: Here is some dialogue on a technical subject from a researcher who was experimenting with using the model to help debug an engineering issue: When asked to parrot writing styles, the model is more likely to make potentially harmful, explicit, or offen- sive statements: In this case we chose a human reply that takes the conversation in a subtly dark direction, and the model rebuts the argument: 7 Discussion We have shown that it’s possible to use reinforcement learning from human feedback to train language models that act as helpful and harmless assistants. Our RLHF training also improves honesty, though we expect other techniques can do better still. As in other recent works associated with aligning large language models RLHF improves helpfulness and harmlessness by a huge margin when compared to simply scaling models up. (スティーニョン et al. 2020年) Thoppilan et al., 2022, Ouyang et al., 2022, ナカノ et al. 2019年 ミニク et al. 2024年 Our alignment interventions actually enhance the capabilities of large models, and can easily be combined with training for specialized skills (such as coding or summarization) without any degradation in alignment or performance. Models with less than about 10B parameters behave differently, paying an ‘alignment tax’ on their capabilities. This provides an example where models near the state-of-the-art may have been necessary to derive the right lessons from alignment research. The overall picture we seem to find – that large models can learn a wide variety of skills, including align- ment, in a mutually compatible way – does not seem very surprising. Behaving in an aligned fashion is just another capability, and many works have shown that larger models are more capable finetune with greater sample efficiency そして、忘れることから大きな苦しみを受けないでください。 Although we did not demonstrate it directly, we also expect that RLHF alignment training can be mixed with or precede train- ing for other objectives; this might be relevant in the future in order to avoid the production of intermediate, unaligned AI systems. [Kaplan et al., 2020年、 Rosenfeld et al., 2019, ブラウン et al. 2020年、 (ハンガー et al. 2020年) Askell et al., 2021], (ラマシュ等、2022年) われわれは、役に立つことと無害性の間に明確な緊張が見つかり、役に立つことのために完全に訓練されたモデルは、HHのために訓練されたモデルに優先されるが、役に立つことだけを評価するときである。われわれは、これは部分的にデータ収集プロセスの微妙さによるものであると信じるが、われわれは、有害な要求(すなわち「人質交渉者」である)に対処する方法についてのデータ教育モデルをめったに収集するが、それらをどのように回避するかについてのみである。また、われわれはまた、少なくとも好みモデルのレベルでは、役に立たない緊張が大きくなり、より有能になるにつれて減少することを発見した。 Large generative models have been referred to as ‘foundation models’ これらのモードは研究のための非常に興味深いオブジェクトですが、さらなる精密化なしに、彼らは有害な行動を示すことができます。私たちの仕事は、調節トレーニングは、その有用性と多様性を損なうことなく基礎モデルに組み込まれることができると示唆しています。 [Bommasani et al., 2021]. 7.1 Limitations While we believe our results present a promising picture for the alignment of existing language models, work on this subject remains in an early stage, and has a number of limitations. As was also emphasized by the authors of 我々は、同調に関する我々の作業を継続的なプロジェクトとして見る。 was step zero, and this is step one. [Thoppilan et al., 2022], (アスケル et al. 2024年) We’ve pragmatically defined an aligned assistant as an AI that is helpful, honest, and harmless. We are op- timistic that at present capability levels, the techniques we have discussed here provide a reasonable approach to achieving helpfulness and harmlessness. However, although our techniques improve model honesty, we believe we are just scratching the surface of that problem, and that other techniques may more efficiently and effectively produce honest AI models. 18 Here we have essentially focused on the average-case behavior of our models. However, even if we were convinced that our models were HHH in expectation, a clear next step would be to attempt to study and eliminate bad behaviors (especially harmfulness) even in the worst case. We have not addressed this question of robustness here, but hope to study it in the future (approaches such as AIシステムが進歩し、展開中に配布の転換に直面するにつれて、より緊急になるだろう。 [Perez et al., 2022] たとえば、私たちの大型RLHFトレーニングモデルは、ほぼすべての能力評価で単純なLMよりも優れているが、本当に役に立つモデルのゼロショットパフォーマンスは、並行しないモデルの数ショットパフォーマンスに等しいと期待できる。 we close a bit less than half of this gap (Figure 私たちはまた、RLHFの精密コードモデルがバグコードを含むプロンプトにさらされた場合に比較的優位性があるかどうかを簡潔に調査しました。 but we did not find any benefits there. One would hope a fully aligned model would do its best to write correct code, even when given a buggy prompt. [Lin et al., 2021] 5 ) [Chen et al., 2021], We also harbor a general concern that perhaps our techniques only render models aligned ‘on the surface’, and that they still harbor harmful biases or other tendencies that may surface in more subtle contexts. We found that RLHF models have a more positive sentiment towards all racial and religious groups, which seems promising, but does not necessarily indicate that biases have been reduced. And with respect to gender, we found that RLHF model biases are very strongly correlated with the bias of the underlying language models. That said, further work will be required to understand if this is a limitation of RLHF as a technique, or of our particular HH datasets. In any case, we likely need to build more subtle and comprehensive evaluations that include multi-turn dialogue, as this is an area where humans will likely use the models, and it’s also a place where it’s inherently more difficult to measure performance against subtle objectives such as bias and fairness. On a much more practical level, we do not have much experience applying RL techniques to large generative models. Experienced AI practitioners know that there are a large variety of tweaks and tricks that require experimentation to identify, and that can majorly improve the stability and performance of training. We have encountered some stability issues with RL, and although we performed some rudimentary hyperparameter scans, we expect that with more experience and study we could do better. We also did not explore variations in online training, such as literally updating a single PM or RLHF model; rather we retrained these models from scratch on each iteration. Another direction for exploration is to use a non-trivial function of PM scores as the RL reward, distorting the score distribution to e.g. focus more on discouraging bad behavior rather than rewarding good behavior. In summary, there are many future directions to explore for improving RLHF. 最終的な懸念は、AIモデルがますます能力を発揮するにつれて、我々が採用した技術のような技術が引き続き適用されるかどうかである。 goal, which is to establish a set of simple and universal techniques この目標を達成することができると仮定すると、次のステップの1つは、研究者間の合意を構築し、技術がAI能力とどのようにスケールするかを含むより深く調和を理解することです。 initial 19 Another essential step will be to use this baseline as a point of departure for exploring other techniques that can better-address more advanced use cases and more speculative failure modes. New ideas and techniques can then be pragmatically compared with existing methods, and then incorporated into standard practice if they yield further improvements in safety and robustness. Our view is that the most relevant problems and the most creative and effective alignment techniques will be identified and developed through research on concrete AI systems. As we saw in Section we are already encountering examples that point to the limitations of human feedback, and so we need to begin to develop other methods. 6.1位 7.2 Alignment Data as a Public Good この作業では、コミュニティーワーカーの常識が役に立つものと無害なものを定義することを可能にしました。これは、「技術的な調和」という問題、すなわち、特定の技術がAIモデルをより有用で無害なものに訓練するために用いられるかどうかという問題を調査するのに十分でした。 This question should not be the provenance of researchers only. That said, without a clear specifi- cation for the format and type of ‘alignment data’ most relevant for AI training, it has been difficult for anyone other than researchers to gather the information needed to train safe and beneficial AI sys- tems. However, recently several projects (including ours) have used similar methods to teach AI models complex human preferences, and we have also found ランクされた比較スケールに基づく優先モデリングは、他の多くのテクニックよりも優れている。 [Stiennon et al., 2020, Ouyang et al., 2022, Nakano et al., 2021] [Askell et al., 2021] 可能なアプローチの1つは、倫理的、法的、文化的専門知識を持つ独立した組織が、AI行動に対する人間の好みを表現する非常に高品質なデータセットを作成することである(比較を通じて)。そのような組織はまた、新しいガバナンス構造を使用することができ、より大きな集団の社会的関与者が、AI行動の基準を確立するために役割を果たすことができるように、AI行動の基準を作成し、調節するための決定に役割を果たすことができるように、新たなガバナンス構造を使用することもできます。AI言語モデルの急速な改善のために、私たちは、これらのデータセットが人間レベルで優先順位を暗号化する場合、最も価値があると予想します。 私たちの研究は、AIと人間の価値観の調和に関連する公開的に利用可能な研究データセットと評価から恩恵を受けました。 and we plan to release our preference modeling data for others to use in their research. Unfortunately, this does not seem to be a standard practice among alignment researchers, as evidenced by some recent work. While we agree that LLMs themselves can be used for harm, it seems that no such argument can be made for alignment data. [Stiennon et al., 2020, Hendrycks et al., 2021a], AIシステムがより強力になり、より幅広く展開されるにつれて、エラーや誤解のコストは膨大に増加する可能性があります。我々は、高度なAIシステムからの潜在的なセキュリティ障害を説得力を持って解決する唯一の方法は、深い専門知識を持つ研究者コミュニティを構築することであると信じています。 One ostensible reason for secrecy is that organizations may use data from users to develop alignment datasets, and then justify not sharing the datasets on the grounds that it violates user privacy. This is a challenging issue that requires organizations to think about how to reconcile commercial priorities with the need to create a ‘safety commons’ for the community. If alignment becomes interlinked with the concept of commercial moats, that could reduce the overall net level of safety of the AI ecosystem. Therefore, we believe that datasets developed for alignment should be kept separate from commercial data, and should be openly shared to advance research on safe and beneficial AI. 7.3 Broader Impacts 我々は、AIシステムが同時に、パフォーマンスコストなしでより安全かつ有用なものにすることができるという説得力のある証拠を提供することを希望する。上記のように、我々は、どの価値観が受け入れ可能で受け入れられないAI行動を定義するかについて、大いに無知的である。 AI technologies are dual-use, meaning they can be used beneficially and otherwise. We have found the ef- fectiveness of preference modeling and RLHF striking (in our research and others’), and believe there’s very legitimate concern that these techniques could be used for censorship, fraud, and misinformation. Straight- forward commercial use-cases also seem worrisome, especially if optimization for objectives like user en- gagement and persuasion are mixed together. At the most naive level, if you can optimize for ‘harmless’ then you can ‘flip the sign’ and generate harmful systems. また、独自に役立つように訓練されたシステムは、有害な目的のために使用しやすくなり、システムがより強力になるにつれて、有害な可能性を直接抑制することがますます重要になることを示唆しています。 20 たぶん、この作品の最も幅広い影響、そして、コントロール可能で人間のような言語の世代の一般的な発展と拡散 will be cultural. In Figure we used an Elo scale, essentially the chess rating system, to compare and evaluate natural language assistants, and we even included comparison to human writers. This kind of comparison risks trivializing the importance of language, which is certainly not just a game, but the core medium of culture and society. While seeking to align increasingly capable AI systems feels like a robustly good action, how and when to deploy these systems poses more challenging questions – culture is fundamentally a human enterprise, but large-scale generative models hold the possibility of magnifying and minimizing different parts of human culture in unpredictable and opaque ways, which could have broad downstream influences. [Ganguli et al., 2022], 1 Acknowledgments Acknowledgments We thank Sam Bowman, Paul Christiano, Jacob Hilton, Jan Leike, Ethan Perez, and Jeff Wu for helpful feedback on the draft. We thank Daniela Amodei, Jamie Kerr, Jia Yuan Loke, Rebecca Raible, and Tim Telleen-Lawton for support with the project. Author Contributions 著者貢献 Yuntao Bai performed most of the experiments on RLHF and many of the preference modeling experiments. He made major contributions to experimental design, measurement, and evaluation of model performance and behavior. He helped to write the paper. Andy Jones and Kamal Ndoussse built the infrastructure for RL training of large language models. They also built associated plotting and monitoring systems and implemented the PPO algorithm. They helped with the design, implementation, and debugging of RLHF. Amanda Askell はモデル評価を設計し、プロの作家からサンプルや評価を収集し、データ収集の品質と量を改善するためのシステムを構築し、ジャレッドとジャクソンと関連する評価に協力しました。 Anna Chen は一般的な RL と RLHF 実験に協力し、研究設計に貢献しました。 Nova DasSarmaは、潜在的なクラスターインフラストラクチャを管理し、大規模なRLトレーニングと人間のフィードバック収集を可能にしました。 Dawn Drain はコードモデルをトレーニングし、コーディング評価に Saurav と協力しました。 スタニスラフ・フォートはOOD検出と、有益なデータと有害なデータサンプルに対する外部曝露の研究と分析を行いました。 Deep Ganguli はレッドチームのデータ収集と設計を率いており、ジャクソン、リヤン、アマンダ、ベンと頻繁に協力し、ジャレッドと協力して社会的影響評価を設計し、実施し、モデル評価を一般的に支援しました。 Tom Henighan は、ベースの言語モデルのプレトレーニング、データセットの作成、プロジェクトのいくつかの段階でクラスターの管理に協力しました。 ニック・ジョセフは、大規模な言語モデルと好ましいモデルの効率的なトレーニングのためのフレームワークを設計し、構築するのを助けました。 Saurav Kadavath designed and conducted experiments on helpful/harmless dataset mixing. Saurav also ran RLHF training on code models, with support from Yuntao, and ran coding evaluations in collaboration with Dawn. He also ran the majority of the natural language evaluations of basic and RLHF-finetuned models, and helped with RLHF training generally. He helped to write the paper. Jackson Kernion led human feedback crowdworker evaluation and management, and helped to build and maintain the feedback interface. He also helped with data analysis and collaborated on model evaluations. He ran most of our model comparison experiments. トム・コナーリーは、特に迅速かつ効率的なサンプル採取の技術を支援し、シェア・エル・ショウクは、プレトレーニング研究とデータセット構築を支援しました。 Nelson Elhage contributed significantly to pretraining and to engineering vision. Zac Hatfield-Dodds は、コードベースのメンテナンスとエンジニアリング、特に迅速かつ効率的なサンプリングを支援しました。 Danny Hernandez contributed to pretraining and especially to dataset design. Tristan Hume helped with streamlining our infrastructure. Scott Johnston helped with pretraining research. Shauna Kravec contributed to the development and use of our RL systems, and collaborated on RL research. Liane Lovitt helped with red-teaming, and in particular with designing the interface. Neel Nanda は、調和のための研究議論と優先事項に貢献しました。 Catherine Olsson helped advise on human feedback data collection, and contributed advice on alignment and evaluation. Dario Amodei advised the project and led efforts to build and test the RL infrastructure and ML. Tom Brown led engineering efforts, including efficient pretraining, sampling, and the stability and design of RL systems. Jack Clark led societal impacts efforts and advised the project, including on various evaluations. Sam McCandlish led pretraining efforts and advised the project. クリス・オラは、調節に関する議論に協力し、私たちの研究と評価インフラトラクチャーに貢献しました。 Ben Mann led the design and construction of the human feedback data collection interface and the underlying infrastructure. He also helped lead crowdworker management, and he provided engineering support for the project as a whole. He also contributed to pretraining and cluster management. Jared Kaplan は、研究設計、エンジニアリング、実験、評価を含むすべての側面に協力し、プレトレーニングに貢献し、評価インフラストラクチャの構築に貢献しました。 A 監督トレーニングの詳細、分析、および評価 A.1 Context Distillation For context distillation, we follow the prescription from 具体的には、まず以下の方法でデータを生成します。 [Askell et al., 2021]. 1.我々は「HHHプロンプト」 (すなわち、役に立つこと、無害性、誠実さを誘発するように設計されたプロンプトのセット) をテキストのセクションに準備し、テキストの50%は私たちのプレトレーニングデータセットから来て、50%はStackExchangeデータセットから来ている。 2. For both datasets, we then perform forward passes with a basic pretrained 52B model, and record the top 50 log-probabilities and their indices (within the vocabulary) for the tokens following the prompt. We store the log-probs, indices, and tokens together as a small new dataset. 3. To perform context distillation finetuning, we pass the tokens from our new dataset through models (of all sizes), and define the loss as the KL divergence between the stored log-probs and the predic- tions of the model undergoing finetuning. For each token, we use a 51-category distribution, with the 51st category covering the total probability from all the tokens other than the top-50. We show learning curves for context distillation in Figure 我々は32セクションのバッチサイズを使用し、プレトレーニング学習率の0**05倍の学習率を使用し、我々は蒸留中に線形的にゼロに分解します。 30. A.2 Preference Modeling 当社の優先順位モデルは、比較データで訓練されており、各データポイントは、 and a pair of The prompt is a multi-step dialogue between human and model that always begins and ends on the human side, and each response is a continuation of the dialogue. 例えば、人間とモデルとの間の多段階的な対話は、常に人間側から始まり、終わり、それぞれの反応は対話の継続である。 プロンプトは対話の最初の5ステップで構成され、答えは青いボックスに表示されます。PMはそれぞれの回答の終わりにスコアを割り当てます。PMは最終的な回答の質を評価するために訓練されているが、会話の完全な文脈はモデルに提供されます。 prompt responses 6、 We train scans of PMs ranging from 13M to 52B parameters. All PMs go through three phases of training: (1) language model (LM) pre-training on a large language corpus, (2) preference model pretraining (PMP), and (3) finetuning on human feedback. LM pre-training details, including choice of hyperparameters and datasets, are explained in Appendix A of our previous work [Askell et al., 2021]. For PMP, we use learning rate of 0.1 relative to LM pretraining, and train on a mixture of comparison data made from StackExchange, Reddit, and Wikipedia. Data preparation and labeling are explained in Appendix C1 We train with context size of 1024 tokens. [Askell et al., 2021]. For human feedback finetuning, we use learning rate of 0.01 relative to the LM pretraining. We use context size of 1024 tokens, except for the ‘online’ model described in Section where we trained with 2048, which may help stabilize RLHF on long contexts. 4.5, For both PMP and human feedback finetuning, we append a special ‘end-of-context’ token at the end of each sample, such that the PM score is predicted directly on top of this token. As explained in Appendix C.4 of this appears to improve PM performance. [Askell et al., 2021], すべての段階では、過剰装備を軽減するために、1回のイーテレーションのみをトレーニングします。 A.3 Scaling of PM with Model and Dataset Size 主要な問題は、モデルサイズとデータセットサイズで優先モデリングのスケーリングのパフォーマンスをどのように行うかです。これは実用的な質問に関連しています - より大きなデータセットの収集に投資するべきですか、それともより大きなモデルのトレーニングに投資するべきですか。 We seem to find more predictable scaling when training only on our helpfulness dataset, likely because the red-teaming data truly comes from a distinct distribution. Accuracy learning curves can be seen on the left of Figure We find that accuracy can be roughly fit by 32. where is the number of parameters in the PM and is the size of the dataset. P D However, the results when training on another preference modeling data distribution look quite different, as seen on the right in Figure Note that there appears to be a sort of discontinuity in behavior between 200M and 13B parameters. Perhaps this is related to the fact that the data was generated by a model with 6B parameters. 32. B Details, Analysis, and Evaluations of RLHF B.1 Training Setup B1 トレーニング ここでは、RLHFトレーニングについていくつかの詳細を議論します. We initialize our policies on context-distilled models, which are explained in A.1. 私たちは、人間のフィードバックにフィニッシュされたPMに比べてスコアを最大化するプロンプトのデータセットへの応答を生成するためのポリシーをトレーニングします。プロンプトのデータセットは、PM比較データセットのトレーニング分割から単に各カップルの応答を削除することによって得られます。 プロンプト内で複数のステップ対話を許可することを思い出してください(常に会話の人間側から始まり、終わる)、しかし、プロンプトごとに1つの応答を生成するためのポリシーをトレーニングするだけです。 将来の仕事では、複数のステップを生成するためのポリシーをトレーニングすることを計画しますが、これは会話の人間側を生成する別のモデルを必要とし、会話の人間側を模 私たちは様々なハイパーパラメータスキャンを実行し、プレトレーニングに比べて 0.01 の学習率を使用し、KL 報酬因子 λKL = 0.001 (4.1), PPO クリッピング = 0.2,割引因 γ = 1,そしてエントロピーボーナスなしで終了しました。さらに、PPO では、同じサンプル K 回数([Schulman et al., 2017]のアルゴリズム1を参照)、より高い K は通常より安定した結果をもたらします。我々は RLHF スキャンのための K = 1、強度研究のための K = 2 (セクション 4.2)、および「オンライン」 RLHF のための K = 4 (セクション 4.5) を繰り返しました。我々はまた、強度の研究のための 32 B.2 More on Robustness Studies B.2 More on Robustness Studies In Figure we compare the すべてのポリシーサイズおよびすべての sizes. The main observation here is that the slope of the graph increases with respect to size, thus suggesting that larger 政治の実績を大きく区別することができる、つまり、 , in accordance with calibration studies in セクション 33, PMテスト test PM PMテスト test PM larger preference models are more robust 3.3. 最後に、我々はここで一瞬にして、これまで無視してきた問題を解決し、すなわち、異なる優先順位モデルのスコアを直接比較すべきではない、スコアの絶対値は意味がないので、相対的なスコアだけが意味があるということである。 procedure. We make a held-out dataset, consisting of several thousand samples, and subtract from each preference model score its mean score on this dataset. We apologize that unrelated plots from different sections may have used different held-out datasets for mean-removal. mean removal B.3 Details of ‘Online’ RLHF B.3 Details of ‘Online’ RLHF We give some more details on our ‘online’ RLHF policy discussed in Section This policy and its PM were trained on all the helpfulness and harmlessness data we had near the completion of this paper. We re-iterated each sample = 4 times to improve stability, and sampled a maximum of 128 tokens per response. 4.5位 K [Schulman et al., 2017] Throughout training, we periodically evaluate the より具体的に、プロンプトと応答の両方からなる比較を考慮すると、より良い応答のためのポリシーの平均ログ-p がより高い場合に 1 を正確に割り当て、その他の場合は 0 を割り当てます。 for various comparison datasets. In particular, we find that mean log-p accuracy of the policy isn’t as high as (i.e., fraction of comparisons on which the PM assigns a higher score to the better response), possibly suggesting room for further improvements to our RLHF pipeline. mean log-p accuracy 34 PM accuracy B.4 Robustness of ‘Online’ RLHF B.4 Robustness of ‘Online’ RLHF For our final online models, we had crowdworkers compare a variety of model snapshots from RLHF training, in order to better understand the robustness and general performance of our training process. In Figure (右)、RLHFトレーニング中にオンラインモデルのEloスコアを表示します。 15 In particular, we compare Elo scores established empirically from crowdworker preferences (i.e., in the figure), and Elo scores predicted by our preference model (i.e, ) RLHFトレーニング中に. 後者のために,我々は,それぞれのスナップショットからの応答を一連の持たれたプロンプトでサンプルし,PMスコア(その後Eloユニットに変換される)を評価します. 我々は,天才的なPM予測が実験的Elosを大幅に過大評価していることに気付きます. これは以下の事実の組み合わせによるものです: Crowd- Worker Preferences(労働者優先順位) Naive PM Predic- tion 1. During crowdworker testing, each step of the conversation is written by one of the two models being tested. However, when evaluating a RLHF snapshot on held-out prompts, the policy only writes one response at the end of a pre-existing conversation (which had been previously generated by other models, as discussed in Appendix This leads to distributional shift between the conversations. B.1). 2. Elo and PM scores may not actually be transitive, as they involve the collapse of pairwise compar- isons onto a single line. For example, if PM scores 満足 − = 2 and − = 2, even if those are well-calibrated scores, the implication that − = 4 may not be correctly calibrated, and we would naturally expect instead − 4. a, b, c a b b c a c a c < 3.PMの堅牢性の欠陥により、PRのRL政策のサンプルに対する好みが、真の人間の好みに比べて誤算される。 To explore these effects further, in Figure we show Elo scores corresponding to four different measure- ments: 35 • Naive PM Prediction: RLHFトレーニング中に記録されたPMスコア(Eloユニットに翻訳)は、一連のHold-Outプロンプトを使用します。 • Mean PM Score on Crowdworker Data: クラウドワーカーの好みに基づいてEloスコアを計算するために使用される実際のクラウドワーカーの会話の平均PMスコア。 • PM Ranking on Crowdworker Data: One can try to distinguish robustness failures from miscal- ibrated PM scores vs PM ここでは、これらのモデルスナップショットを比較するために使用されたクラウドワーカーデータでPMを評価し、モデル比較のための「勝利率」を得る。 , and then we the Elo scores based on the PM’s choices. rankings PMによると、 recompute • Crowdworker Preferences: We straightforwardly compute Elo scores based on crowdworker prefer- ences among model snapshots. So we see that the PM score vs PM rankings distinction does not make a significant difference in terms of robustness. However, the distributional shift between the held-out prompts and the actual crowdworker conversations was very significant, and explains a significant proportion of the discrepancy between RLHF learning curves and the Elo scores as measured from crowdworkers. B.5 Crowdworker Comparisons and Elo Scores B.5 Crowdworker Comparisons and Elo Scores(クラウドワーカーの比較とエロスコア) Here we briefly describe how we test crowdworker preferences of our models, and how Elo scores are es- tablished. For pair of models and , we ask crowdworkers to engage in text-based, back-and-forth con- versations with the models. At each conversational step, two responses are generated, one from each model, and the worker chooses the response they prefer, and the conversation continues. Each choice the worker makes counts as a ‘win’ for the preferred model, giving ‘win counts’ , respectively. In cases where a worker is unsure about whether one response is better, we throw out such comparisons in both PM and RLHF training, and crowdworkers comparison evaluations. A B NA, NB Recall that, given Elo scores EA, EB, respectively, the log-likelihood for the win counts is given by where rA,B = (log 10/400)EA,B ≈ EA,B/174. For an ensemble of comparisons between various models, we estimate Elo scores and their errors by maximum likelihood estimation. In some cases one of the models uses , meaning that it generates samples, evaluates all of them using a preference model, and shows the user the top-scored sample. Elo scores for such models are shown in Appendix In this case, we cannot stream the sample, so instead we make the workers wait until the sample is completed. When testing a rejection sampling model against a non-rejection sampling one, we only show the samples when they’ve both been completed, even if the latter sample could’ve been streamed, to mitigate bias. rejection sampling k B6 B.6 Elo Scores for Rejection Sampling Models B.6 Elo Scores for Rejection Sampling モデル In Figure we show helpfulness Elo scores for a 52B context distilled model with rejection sampling (utilizing a 52B preference model trained on pure helpfulness) for = 1*,* 4*,* 16*,* 64, showing that higher values of clearly perform better. Note that the context distilled model and the preference models discussed here were trained during an earlier stage of our research with different datasets and settings from those discussed elsewhere in the paper, so they are not directly comparable with other Elo results, though very roughly and heuristically, our online models seem to perform about as well or better than = 64 rejection sampling. Note 36 k k k that = 64 rejection sampling corresponds to = log(64) ≈ 4*.*2. k DKL B.7 Stack Overflow Results We can also evaluate our language models directly given a corpus of paired good and bad responses, such as answers to StackOverflow questions. In We evaluate the difference in average log-p between popular (i.e., highly upvoted) and unpopular answers, showing that RLHF models consistently assign a higher difference, suggesting that they are more capable of distinguishing answer quality. RLHF モデルが一貫してより高い差を割り当てていることを示す。 We plot the language modeling loss (i.e., mean log-prob) on the good and bad answers separately, rather than their difference. We find that the RLHF models obtain worse loss. This is most likely due to optimizing a different objective rather than pure language modeling. 我々は、RLHFモデルがより悪い損失を得ることを発見します。 37b 37A B.8 Further Analysis of RLHF on Code-Model Snapshots B.8 Further Analysis of RLHF on Code-Model Snapshots セクションで議論したように RLHF improves performance of base code models on code evals. In this ap- pendix, we compare that with simply prompting the base code model with a sample of prompts designed to elicit helpfulness, harmlessness, and honesty, which we refer to as ‘HHH’ prompts. In particular, they contain a couple of coding examples. Below is a description of what this prompt looks like: 5.3, Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. It also tries to avoid giving false or misleading information, and it caveats when it isn’t entirely sure about the right answer. That said, the assistant is practical and really does its best, and doesn’t let caution get too much in the way of being useful. -- ... (we include several short example conversations using the normal Human: ... Assistant: ... format.) -- Human: Can you help me write this Python function? I’ve already written the function’s signature and docstring, but I’m not sure how to write the function’s body. It starts like this: Assistant: Sure thing, here you go! I’ve tested this function myself so I know that it’s correct: Figure contains results on HumanEval when the HHH prompt is included. We see that the HHH prompt improves performance more significantly than RLHF across many values. 38 pass@k B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests 私たちの新しく提案された簡素化された相対マハラノビス距離は、我々がテストしたすべてのモデルサイズのすべての層から抽出された有用性入力からの無害性入力のOOD検出の標準マハラノビス距離を上回ります。 Simplified Relative Mahalanobis distance 39. OOD探知器を外部配布のいくつかの例にさらすことは、最初に提案されました。 suggested using a single class representing the OOD examples. [F 強力なOOD探知機さえ大きな利点を享受していることが示されました。 Few-shot outlier exposure [Hendrycks et al., 2018]. [Thulasidasan et al., 2021] トップ > トップ > 2021年 Following the procedure from [F we apply a single layer linear classifier on top of the action-vation vectors, while the rest of the language model is frozen. 言語モデルの残りの部分は凍結されます。 randomly drawn examples of the training set of the out-distribution (harmlessness data), we create a new binary classification problem, where the inputs are the combination of the full train examples of training set of the in-distribution with a target class of 0, and train*//M* copies of the ターゲットクラス1のアウトディストリビューションの例 1. これにより、インディストリビューションとアウトディストリビューションの両方の例の均等な表示が確保され、より良い学習につながります。線形分類器を訓練すると、そのクラス1の出力をOODスコアとして使用します。 トップページ > 2021年(昭和20年) M N N M 23. B.10 Gender Bias Evaluation Details B.10 Gender Bias Evaluation Details For this evaluation, we followed and used their list of 76 occupations and 12 pairs of gender terms: [Rae et al., 2021] • Occupations: CEO, accountant, administrator, advisor, analyst, appraiser, architect, assistant, atten- dant, auditor, baker, bartender, broker, carpenter, cashier, chef, chemist, chief, cleaner, clerk, cook, counselor, designer, developer, dietitian, dispatcher, doctor, driver, editor, educator, electrician, engineer, examiner, farmer, firefighter, guard, hairdresser, hygienist, inspector, instructor, investigator, janitor, lawyer, librarian, machinist, manager, mechanic, nurse, nutritionist, officer, painter, par- alegal, paramedic, pathologist, pharmacist, physician, planner, plumber, practitioner, programmer, psychologist, receptionist, salesperson, scientist, secretary, sheriff, specialist, supervisor, surgeon, tailor, teacher, technician, therapist, veterinarian, worker, writer 男性の性別:男の子、兄、父、夫、男、花婿、男、男、男、父、息子、紳士、ボーイフレンド 女性の性別用語:女の子、姉、母、妻、妻、花嫁、女の子、ガール、女の子、母、娘、ガール Human: Can you help me fill in the missing word in a sentence? The sentence is: For work, the {gendered term} is a Assistant: Sure thing, here is your sentence with the missing word filled in: For work, the {gendered term} is a {occupation} We then evaluated the probabilities of , given gender terms. With this formulation, the larger LMs, context-distilled models, and RLHF models all assigned about 2-8% of their probability mass to one of the 76 occupations listed above when completing the sentence. 職業 We also slightly modified the metric from [Rae et al., 2021] in order to estimate errors. So to compute bias scores, for each occupation O we defined and then we define the overall bias score as the mean of the absolute value of , ranging over occupations: BO where represents occupations while ジェンダー・ジェンダー・ジェンダー・ジェンダー・ジェンダー・ジェンダー・ジェンダー・ジェンダー・ジェンダー・ジェンダー・ジェンダー 76と、 = 12. With this setup, we are sensitive to whether bias is uniform across all pairs of gendered terms (for any given occupation). As a concrete example, if it were the case that a model associated ‘lawyer’ more strongly with ‘boy’ than ‘girl’, but also more strongly with ‘wife’ than ‘husband’, this would increase the error in our measurement, while the bias effects would tend to cancel out. O gf , gm NO NG 結果は図で示す。 where we observe that RLHF models have both larger bias scores and larger errors. We believe this has a very simple explanation – RLHF models have much lower-entropy output distributions. As a rough, heuristic explanation, a model with a lower-entropy output will behave somewhat like a plain LM evaluated at a temperature 1. The metric and error from equation will typically scale as 1*/T* , so it should be expected that models with lower entropy distributions will have proportionally larger bias scores and bias errors. We show these effects explicitly in Figure by evaluating bias scores with context distilled models at ≈ 0*.*6, and showing that they are quite similar to RLHF bias scores. 40, T」 B.3 40 T 読者は、RLHFモデル偏差がモデルサイズとともに増加しているように見えるかもしれません。これは、RLHFトレーニングがモデル配布のエントロピー(または「効果的な温度」を徐々に減少させる傾向があるためです。 Cサンプル from PALMS, LaMDA, and InstructGPT Prompts これらは、当社のベストオンライン HH モデルから各プロンプトに対して 17 件の回答をサンプル化し、それらを適切な好みモデルを使用してランク付けし、その後、プロンプトを表示します。 non-empty response ac- cording to the PM. (Occasionally the model writes the "end of conversation" token in response to sensitive queries.) So to a large extent, these samples are not ‘cherry-picked’, and should be representative of average- case behavior. In the case of LaMDA 「InstructsGPT」 これらのモデルから相応のサンプルを含む。 median [Thoppilan et al., 2022] [Ouyang et al., 2022] C.1 PALMS Sensitive Questions C.1 PALMS 繊細な質問 繊細な質問から取り上げられています。 (ソライマンとデニソン、2021年) これらの質問のいくつかには、敏感な資料(設計により)が含まれていることをご注意ください。 C.2 InstructGPT Prompts われわれは読者たちに、インストラクトGPTのサンプルの一部を警告する。 たぶん桜が展示されていたのだろう。 , and so may be unrepresentative. Thus in some cases it will be unfair to compare our median samples to those from InstructGPT. [Ouyang et al., 2022] poor model behavior, on purpose C.3 LaMDA Prompts C.3 LaMDA Prompts D Details on Data Collection and Crowdworkers D.1 Overview D.1 Overview For the purposes of assigning tasks, we divided our crowdworkers into two groups. A group of around 30 ‘select’ workers were sent high priority tasks on a regular basis. Roughly half of these workers were hired through Upwork. The other half were selected from the general population of US-based MTurk workers with a ‘Masters’ qualification. The second group of workers came from the same general population of MTurk workers, and were primarily given red-team tasks on a less regular schedule. We tried to ensure that all workers were paid significantly above the minimum wage in California, adjusting rates for tasks that took longer (for example with models that use rejection sampling) in the case of MTurk workers, who are paid by task rather than at an hourly rate. We frequently issued bonuses to MTurk workers who encountered technical problems with tasks (as they are not paid an hourly rate). Among our select workers, MTurk workers usually accounted for 80-85% of comparison data collected in a given week, compared to 15-20% for workers hired through Upwork. Although the size of these groups were similar, MTurk workers tended to opt in to more work and their pay structure incentivized quicker conversations. Slackを通じて、私たちはこのチャンネルを使用して新しいタスクを発表し、ガイドラインを提供し、彼らが登場したときにグループと困難なエッジケースを議論しました。 Our workers alerted us to bugs and performance issues as they ran into them. 43. We sent both groups of workers a demographics survey, and the results are shown in figure 調査の再スポンサーは匿名であったため、我々は人口統計情報に加えて個人情報を収集しませんでした。 44. D.2 Instructions and Interface D.2 Instructions and Interface We display basic task instructions in a pop-up dialog when first loading the interface, and these instructions remain available throughout the interaction. These instructions for the ‘playground’ and ‘red team’ tasks can be found in figure 遊び場のタスクについては、より詳細な例を含む拡張された指示を含む別ページにリンクし、その抽象は図に示すことができます。 41. 42. 人間のフィードバックインターフェイスは、図で示されています。 オンラインデータ収集プロセス中に、Upworkers のインターフェイスに追加オプションを追加しました。この機能により、モデル応答のいずれかを編集することができました。この機能を使用したとき、編集の比較をオリジナルと保存しました(編集がより良いと仮定しました)、2 つのモデル出力の初期比較ではなく、オンラインデータの 10% 未満に影響を与えました。 6. D.3 Data Quality Measurement Challenges D.3 データ品質測定の課題 一般的に言えば、ヒューマンラベル用のデータ品質保証には、しばしば以下のステップが含まれます。 • 研究者は、小さなサンプルのセットを慎重に検討し、「ゴールデン」ラベルのセットを作成します。 • Human labelers work through a stack of labeling tasks, with some subset of labeling tasks assigned to multiple labelers, and samples from the golden label dataset sent out to everyone. • Researchers evaluate labeler performance by checking their labels against the golden labels, and by checking for inter-labeler agreement The idea is that the golden labels are treated as a source of truth, with labelers incentivized to modify their behavior to better match the golden labels, and novice labelers incentivized to match the behavior of more experienced labelers. オープンエンドの会話は、より豊かなデータセットを収集することを可能にしましたが、いくつかの制御しにくい変数を導入し、騒々しいデータ品質指標を生み出しました。 We did try having crowdworkers review each other’s conversations, by providing ratings for each model response comparison and rating the overall quality of the conversation. But we found that author-rater agree- ment wasn’t a good guide for assessing overall conversation quality. In broad strokes, conversation quality depends on a) conversation topic, b) human writing quality, c) model quality. And we found that, for instance, as conversations got more sophisticated, deciding between model responses got more difficult. As a result, authors that more frequently discussed difficult topics would often get lower agreement scores. And because we were frequently updating our models, model response comparisons were a moving target. Instead of directly assessing our crowdworkers based on these kinds of reviewer-based metrics, we considered using their performance as reviewers as a stand-in for their performance as conversation authors. But when we compared these metrics against our own spot-checks of conversation author quality, we found poor agreement. 我々はこれらの問題を回避し、将来より良い方法を発明することを期待しているが、高度なデータ品質管理なしに我々の結果を達成することができたことは注目すべきである。 NLP Evaluations Formatting and Promptsについて ここでは、Lambada、ARC、MMMLU、HellaSwag、OpenBookQA、TriviaQAのパフォーマンスを評価するために使用した入力フォーマットを示します。 参照 [Amodei et al., 2016] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). セキュリティにおける具体的な問題点 [Askell et al., 2021] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general language assistant as a laboratory for alignment. [Bender et al., 2021] Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? ï¿œï¿œ. In , FAccT ’21, pages 610–623, New York, NY, USA. Association for Computing Machinery. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency [Bommasani et al., 2021] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gespillie, L., Goel, K., Goodman, N. D., Grossman, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M. S., Krishna, R., Kuditipudi, R., and et al. (2021). On the opportunities and risks of foundation models. ABS/2108.07258 CoRR [Borgeaud et al., 2021] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J. W., Elsen, E., and Sifre, L. (2021). Improving language models by retrieving from trillions of tokens. , abs/2112.04426. コロ [Brown et al., 2020] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. [Chen et al., 2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. . arXiv preprint arXiv:2107.03374 [Chowdhery et al., 2022] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., ペラット、M、Lewkowycz、A、Moreira、E、子供、R、Polozov、O、Lee、K、Zhou、Wang、X、Saeta、B、Diaz、M、Firat、O、Catasta、M、Wei、J、Meier-Hellstern、K、Eck、D、Dean、J、Petrov、S、およびFiedel、N。 [Clark et al., 2018] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018年) あなたは質問に答えて解決したと思いますか? try arc, the ai2 reasoning challenge. , abs/1803.05457. ArXiv [Fort, 2022] Fort, S. (2022). 強力なほぼ配布外検出の敵対的脆弱性。 [Fort et al., 2021] Fort, S., Ren, J., and Lakshminarayanan, B. (2021). Exploring the limits of out-of- distribution detection. [Ganguli et al., 2022] Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S. E., Fort, S., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Amodei, D., Brown, T., Kaplan, J., McCandlish, S., Olah, C., and Clark, J. (2022). Predictability and surprise in large generative models. [Guo et al., 2017] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. [Guu et al., 2020] Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020)。 , abs/2002.08909. CoRR [Henderson et al., 2017] Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., and Pineau, J. (2017). Ethical challenges in data-driven dialogue systems. スイス / 1711 09050 コロ [Hendrycks et al., 2021a] Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. (2021a) 共通の人間的価値観とAIを調和させる。 [Hendrycks et al., 2021b] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Stein- hardt, J. (2021b) 大規模なマルチタスク言語理解の測定。 [Hendrycks et al., 2021c] Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. (2021c). mlの安全性における未解決の問題。 [Hendrycks and Gimpel, 2016] Hendrycks, D. and Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. [Hendrycks et al., 2018] Hendrycks, D., Mazeika, M., and Dietterich, T. (2018)。 [Henighan et al., 2020] Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. (2020). Scaling laws for autoregressive generative model- ing. [Hernandez et al., 2021] Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. (2021). 転送のためのスケーリング法則。 , abs/2102.01293. CoRR [Hestness et al., 2019] Hestness, J., Ardalani, N., and Diamos, G. (2019). Beyond human-level accuracy: Computational challenges in deep learning. In , PPoPP ’19, ページ 1–14, ニューヨーク, NY, USA. ACM. Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming [Jiang et al., 2021] Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., and Choi, Y. (2021). Delphi: マシン倫理と規範に向かって。 [Joshi et al., 2017] Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale remotely supervised challenge dataset for reading understanding. 読書理解のための大規模な遠隔監督データセット。 [Kaplan et al., 2020] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020) 神経言語モデルのスケーリング法則。 [Koch et al., 2021] Koch, J., Langosco, L., Pfau, J., Le, J., and Sharkey, L. (2021). Objective robustness in deep reinforcement learning. アブス2105.14111 CoRR [Lakshminarayanan et al., 2016] Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016). 深いセットを使用して単純でスケーラブルな予測不確実性の推定。 [Lee et al., 2018] Lee, K., Lee, K., Lee, H., and Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. [Lewis et al., 2020] Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. イギリス / イギリス / 2005年11401 コロ [Liang et al., 2017] Liang, S., Li, Y., and Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. [Lin et al., 2021] Lin, S., Hilton, J., and Evans, O. (2021). Truthfulqa: モデルが人間の偽りを模する方法を測定する。 [Liu et al., 2021] Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. (2021). On-the-flyコントロールされたテキスト生成と専門家と反専門家。 , abs/2105.03023. CoRR [Liu et al., 2020] Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., and Lakshminarayanan, B. (2020)。 Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. [Menick et al., 2022] Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. (2022). Teaching language models to support answers with verified quotes. [Mihaylov et al., 2018] Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). 装甲のスーツは電気を導くことができますか? オープンブックの質問に答えるための新しいデータセット。 . EMNLP [Nakano et al., 2021] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. (2021). Webgpt: Browser-assisted question-answering with human feedback. バッハ2112 9332 コロ [Nalisnick et al., 2019] Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B. (2019). 深く反転可能な機能を持つハイブリッドモデル。 [Nguyen et al., 2014] Nguyen, A., Yosinski, J., and Clune, J. (2014). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. 深い神経ネットワークは容易に騙される。 [Ouyang et al., 2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) 人間のフィードバックと指示に従うための言語モデルトレーニング。 . arXiv プレプリント arXiv:2203.02155 [Pan et al., 2022] Pan, A., Bhatia, K., and Steinhardt, J. (2022). 報酬誤定の影響:誤定モデルのマッピングと軽減。 , abs/2201.03544. コロ [Paperno et al., 2016] Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and FernÃ¡ndez, R. (2016). The lambada dataset: Word prediction requiring a broad discourse context. [Parrish et al., 2021] Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. (2021). BBQ: 質問に答えるための手作りの偏見基準。 abs/2110.08193 について CoRR [Paszke et al., 2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019) Pytorch: An imperative style, high- performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, 8024～8035 Curran Associates, Inc. (カレーン・アソシエイツ・インク) Advances in Neural Information Processing Systems 32 [Perez et al., 2022] Perez, E., Huang, S., Song, H. F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. (2022). 赤い言語モデルと言語モデルを組み合わせる。 , abs/2202.03286. コロ [Power et al., 2022] Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. アブス2201.02177 CoRR レイ、J・W・ボルジオード、S・カイ、T・ミリカン、K・ホフマン、J・ソング、H・F・アスランディス、J・ヘンダーソン、S・リング、R・ヤング、S・ルーサーフォード、E・ヘンニガン、T・メニク・J・ハイギンズ、Creswell、A・McAleese、N・W・A・Elsen、E・Jayakumar、S・M・Buchatskaya、E、Budden、D、Sutherland、E、S・Huang、S、Uesato、J・Mellor、J・Higgins、I・Creswell、A・McAleese、N・W・A・Elsen、Jayakumar、 X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B. A., Weidinger, L., Gabriel, I., Isaac, W. S., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G. (2021). Scaling language models: Methods, analysis & insights from training gopher. , abs/2112.11446. コロ [Ramasesh et al., 2022] Ramasesh, V. V., Lewkowycz, A., and Dyer, E. (2022). 神経ネットワークにおけるカタストロフィック忘却に対するスケールの影響。 . 学習代表団に関する国際会議 [Ren et al., 2021] Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., and Lakshminarayanan, B. (2021). A simple fix to mahalanobis distance for improving near-ood detection. . arXiv preprint arXiv:2106.09022 [Rosenfeld et al., 2019] Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. (2019). A constructive prediction of the generalization error across scales. ロゼンフェルド, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. (2019). [Sanh et al., 2019] Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. バットの蒸留されたバージョン:より小さく、より速く、より安く、より軽い。 バンクーバー / 1108 / 1108 アーカイブ [Schulman et al., 2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). 近接政策最適化アルゴリズム。 スイス / スイス / スイス / スイス / スイス / スイス / スイス / スイス / スイス CoRR [Solaiman and Dennison, 2021] Solaiman, I. and Dennison, C. (2021). 価値をターゲットにしたデータセットで言語モデルを社会に適応するプロセス(PALMS)。 アブス2106 1032 CoRR [Stiennon et al., 2020] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. (2020). Learning to summarize from human feedback. クライクン、M、レピヒン、D、Qin、J、Chen、D、Xu、Y、Chen、Z、Roberts、A、Bosma、M、Zhou、Y、Chang、 C、Krivokon, I., Rusch, W., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., and Le, Q. (2022). Lamda: 対話アプリケーションの言語モデル アブス2201.08239 コロ [Thulasidasan et al., 2021] Thulasidasan, S., Thapa, S., Dhaubhadel, S., Chennupati, G., Bhattacharya, T., and Bilmes, J. (2021). 避難を使用して分布外の検出のためのシンプルかつ効果的なベースライン。 [Tillet et al., 2019] Tillet, P., Kung, H. T., and Cox, D. (2019). , 10-19 ページ. Association for Computing Machinery, New York, NY, USA. Triton: A Intermediate Language and Compiler for Tiled Neural Network Computations(トリトン:テルド・ニューラル・ネットワーク・コンピュータのための中間言語とコンパイラ) [Weidinger et al., 2021] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W. S., Legassick, S., Irving, G., and Gabriel, I. (2021) 言語モデルによる害の倫理的および社会的リスク。 アブス2112359 コロ [Winkens et al., 2020] Winkens, J., Bunel, R., Roy, A. G., Stanforth, R., Natarajan, V., Ledsam, J. R., MacWilliams, P., Kohli, P., Karthikesalingam, A., Kohl, S., Cemgil, T., Eslami, S. M. A., and Ronneberger, O. (2020) 配布外の検出を改善するための対照的なトレーニング。 [Xu et al., 2020] Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. (2020). Recipes for safety in open-domain chatbots. . arXiv preprint arXiv:2010.07079 [Zellers et al., 2019] Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). Hellaswag: Machine can really finish your sentence? [Zhang et al., 2020] Zhang, H., Li, A., Guo, J., and Guo, Y. (2020). オープンセット認識のためのハイブリッドモデル. [Ziegler et al., 2019] Ziegler, D., Stiennon, N., Wu, J., Brown, T., Amodei, D., Radford, A., Christiano, P., and Irving, G. (2019). Fine-Tuning GPT-2 from Human Preferences. この論文は、CC by 4.0 Deed (Attribution 4.0 International) ライセンスの下で archiv で利用できます。 この論文は、CC by 4.0 Deed (Attribution 4.0 International) ライセンスの下で archiv で利用できます。