Nützliche und harmlose KI: Ausrichtungstraining verbessert die Leistung bei fast allen NLP-Bewertungen

Trainieren Sie einen hilfreichen und harmlosen Assistenten mit Verstärkungslernen aus menschlichem Feedback Die Autoren: Yuntao Bai Andy Jones Kamal Ndousse von Amanda Askell Anna Chen Neues DasSarma Morgendämmerung Drain von Stanislav Fort Tiefe Ganguli von Tom Henighan Nicholas Joseph Saurav Kadavath von Jackson Kernion von Tom Conerly Sheer El-Showk von Nelson Elhage Zac Hatfield-Dodds von Danny Hernandez Tristan Hume von Scott Johnston Shauna Kravec von Liane Lovitt Neel Nanda von Catherine Olsson Dario Amodei Tom Brown von Jack Clark Sam McCandlish Chris Olah Ben Mann von Jared Kaplan Die Autoren: Yuntao Bai von Andy Jones Kamal Ndousse von Amanda Askell Anna Chen Neues DasSarma Morgendämmerung Drain von Stanislav Fort Tiefe Ganguli von Tom Henighan Nicholas Joseph Saurav Kadavath von Jackson Kernion von Tom Conerly Die El-Show von Nelson Elhage von Zac Hatfield-Dodds von Danny Hernandez von Tristan Hume von Scott Johnston Shauna Kravec von Liane Lovitt Neel Nanda von Catherine Olsson Dario Amodei Tom Brown von Jack Clark von Sam McCandlish Chris Olah Ben Mann von Jared Kaplan Abstract Abstrakte Wir wenden Präferenzmodellierung und Verstärkungslernen aus menschlichem Feedback (RLHF) auf finetune Sprachmodelle an, um als hilfreiche und harmlose Assistenten zu fungieren. Wir finden, dass dieses Ausrichtungstraining die Leistung bei fast allen NLP-Bewertungen verbessert und vollständig mit dem Training für spezialisierte Fähigkeiten wie Python-Codierung und Zusammenfassung kompatibel ist. Wir erforschen einen wiederholten Online-Trainingsmodus, in dem Präferenzmodelle und RL-Politiken wöchentlich mit frischen menschlichen Feedbackdaten aktualisiert werden, was unsere Datensätze und Modelle effizient verbessert. Schließlich untersuchen wir die Robustheit von RLHF-Training und identifizieren eine ungefähr lineare Bezie 1 Einführung Wir möchten Techniken entwickeln, um KI-Agenten auszubilden, die hilfreich, ehrlich und harmlos sind. In diesem Papier zeigen wir, dass wir eine relativ hilfreiche und harmlose (HH) natürliche Sprachassistentin durch Sammlung von menschlichen Präferenzdaten und Anwendung der Techniken der Präferenzmodellierung (PMing) und Verstärkungslernen aus menschlichem Feedback (RLHF). [Askell et al., 2024] 1 2. Unser Ziel ist es nicht, zu definieren oder zu verschreiben, was „nützlich“ und „unschädlich“ bedeuten, sondern die Wirksamkeit unserer Trainingstechniken zu bewerten, so dass wir in den meisten Fällen einfach unsere Crowdworkers diese Konzepte so interpretieren lassen, wie sie es für angemessen halten. Wir behandeln Nützlichkeit und Unschädlichkeit getrennt und sammeln unterschiedliche menschliche Vorlieben-Daten-Sets für jeden. In jeder Phase ihrer Gespräche mit dem AI-Assistent werden Crowdworkern zwei mögliche Antworten vorgestellt. Diejenigen, die sich an der Hilfsaufgabe beteiligen, werden angewiesen, die hilfreichere und ehrlichere (d.h. bessere) Antwort zu wählen. Diejenigen, die sich an der roten Teammitgliedschaftsaufgabe beteiligen, werden angewiesen, die schädlichere (d.h. schlechtere) Antwort zu wählen. 2 3 Hilfsbereitschaft und Unschädlichkeit stehen oft im Gegensatz zueinander.Ein übermäßiger Fokus auf die Vermeidung von Schaden kann zu „sicheren“ Reaktionen führen, die die Bedürfnisse des Menschen nicht wirklich erfüllen.Ein übermäßiger Fokus auf hilfsbereit zu sein kann zu Reaktionen führen, die Menschen helfen, Schaden zu verursachen oder toxische Inhalte zu erzeugen. Wir demonstrieren diese Spannung quantitativ, indem wir zeigen, dass Präferenzmodelle, die trainiert wurden, um in erster Linie eine dieser Qualitäten zu bewerten, auf der anderen Seite sehr schlecht (viel schlimmer als zufällig) funktionieren. Zum Glück finden wir, dass PMs, die auf einer Mischung aus beiden Datensätzen ausgebildet wurden, dennoch die richtigen Lektionen lernen und sich bei Bedarf hilfsbereit verhalten können, während sie die höfliche Ablehnung schädlicher Anfragen fördern. Mit Präferenzmodellen in der Hand trainieren wir dann hilfreiche und harmlose Assistenten über Verstärkungslernen, wobei die PM-Score als Belohnung verwendet werden. Wir bewerten sowohl die PM-Leistung als auch die relevanteren Leistungsmerkmale unserer RLHF- purely helpful RLHF-trained models are far easier to red-team, while helpful+harmless models are both very helpful and much less harmful. 1, der Wir finden, dass, wenn RLHF auf große Sprachmodelle angewendet wird, die Antwort scheint ein fast kategorisches Nein zu sein. Wir argumentieren auch, dass man spezialisierte Fähigkeiten mit Alignment-bezogenen Schulungen mischen kann, ohne die Ausrichtung oder Leistung zu beeinträchtigen.In der Praxis sind Aligned-Modelle wahrscheinlich benutzerfreundlicher und implementierbarer als ihre rohen Gegenstücke, was darauf hindeutet, dass es wenig Grund gibt, Modelle zu implementieren, die nicht für die Ausrichtung abgestimmt wurden. 3. 1.1 Beiträge Dialogue Preference Datasets • Wir sammeln getrennte Datensätze zur Hilfsbereitschaft und Unschädlichkeit (d.h. Red-Teaming) hauptsächlich unter Verwendung verschiedener 52B-Sprachmodelle (siehe Abschnitt für Details) in unserer Schnittstelle (Abbildung Crowdworkers führen offene Gespräche mit den Modellen, entweder um Hilfe bitten oder Anweisungen geben oder versuchen, das Modell zu verführen, schädliche Reaktionen zu emittieren, und sie werden gebeten, die hilfreichere Antwort oder die schädlichere zu wählen. Antwort auf jeden Gesprächsschritt. 2 6 ) 4 • Wir sammeln drei Datenabschnitte, einen von unseren Anfangsmodellen, einen mit Ablehnungsproben gegen frühe Präferenzmodelle und einen endgültigen Datensatz, der mit Modellen gesammelt wurde, die mit „online“-Verstärkungslernen aus menschlichem Feedback ausgebildet wurden, den wir ungefähr wöchentlich verbessern. 2.3 Die Alignment with Human Values Has Many Benefits and Essentially No Cost to Performance • Kleinere Modelle erleiden schwere „Alignment-Steuern“ – ihre Leistung bei einer Vielzahl von Bewertungen sinkt nach dem RLHF-Training. mit unseren 13B und 52B RLHF-trainierte Modelle leisten bei Null-Shot-NLP-Bewertungen besser, und das gleiche bei wenigen-Shot-Bewertungen. alignment bonuses 5 • Natürliche Sprache RLHF-Training für HH kann auf Modelle angewendet werden, die zuerst auf Code finetuned wurden, und es auf Bewertungen (vermutlich durch die Verbesserung der allgemeinen Anweisung nachfolgend). Wir finden auch, dass das Mischen von Vorlieben Modell-Training für HH mit der Die Zusammenfassung Es tritt keine Degradation in der Leistung in der HH oder der Zusammenfassung auf.So gibt es keinen Grund, das Ausrichtungstraining nicht mit spezifischeren, wertvollen Fähigkeiten zu kombinieren. improves their programming ability specialized skill [Stiennon et al., 2020] • Es gibt eine , die sowohl auf der Ebene der Präferenzmodellierung als auch der RLHF-ausgebildeten Richtlinien gemessen werden können (Abbildung Da sich jedoch die Modellgröße erhöht, funktionieren PMs auf beiden Verteilungen gleichzeitig besser und werden viel robuster gegenüber den relativen Proportionen nützlicher und harmloser Trainingsdaten. tension between helpfulness and harmlessness 1 ) • Wir zeigen auch, dass man OOD-Erkennungstechniken [F um die seltsamsten und schädlichsten Anfragen abzulehnen (Abbildung mit wenig oder keinem schädlichen Beispiel (Abbildung Ortschaften et al., 2021) 22), 23 ) Scaling, RLHF Robustness, and Iterated ‘Online’ Training • Wir untersuchen Skalierungsverhältnisse für PM-Genauigkeit als Funktion von Modell- und Datensatzgröße und finden ungefähr log-lineare Trends (Abbildung though we encounter some idiosyncrasies (Figures und 7 ) 31 Die 32). • Wir führen Experimente auf der (siehe Bild wo wir unsere Datensätze in die Hälfte teilen und auf jeder Hälfte separate Präferenzmodelle trainieren. Dann trainieren wir RL-Modelle gegen eine PM, während wir mit der anderen bewerten. Wir kommen zu dem Schluss, dass größere PMs robuster sind als kleinere PMs, und wie erwartet, übermäßige Erhöhungen während des RLHF-Training. robustness of RLHF 4 ) • Wir finden das in der ( 0) und die Belohnung ist für einen Großteil der RLHF-Training (siehe Zahlen und wo und Wir erklären, wie diese Beziehung entstehen kann und besprechen mögliche Anwendungen und zukünftige Richtungen. D Die π approximately linearly related 4 13 ) π π • Wir studieren , wo wir unsere Präferenzmodelle und RLHF-Richtlinien wöchentlich aktualisieren und dann diese frischen RLHF-Modelle neu einsetzen, um mit Crowdworkern zu interagieren. und stark verbessert unseren Datensatz, wie von unseren eigenen PMs beurteilt (Abbildung Füllen Sie den oberen Schwanz in Bezug auf Qualität. iterated online training 1 ) 15 ) Um Verwirrungen zu beseitigen und unsere Schlussfolgerungen zu stärken, führen wir zusätzliche kontrollierte Experimente durch (Abbildung Datensatzgröße und andere Hyperparameter festhalten. 16 ) 1.2 Zusammenfassung von Bewertungen und Metriken • die : Wir bewerten unsere Modelle auf MMLU Bastard [P] Griechenland OpenBookQA ARC und trivial Diese Figuren and für vollständige Ergebnisse und Figur In jedem Fall, mit Ausnahme von TriviaQA, 12B und 52B RLHF-trained Modelle besser als Basis LMs. Separat, nehmen wir Python-Codierung Modelle und finetune sie mit natürlichen Sprache RLHF, und dann bewerten sie auf dem Codex HumanEval Diese Figur We also experiment with mixing PM training for HH with summarization als spezialisierte Fähigkeit, und bewerten die resultierende PM-Leistung (Abbildung fanden heraus, dass gemischtes Training die PM-Genauigkeit nicht verringert. NLP and Code Evaluations [Hendrycks et al., 2021b] abgelaufen et al., 2016 [Zellers et al., 2019] [Mihaylov et al., 2018] [Clark et al., 2018] [Joshi et al., 2017] 28 29 3 [Chen et al., 2024] 21. [Stiennon et al., 2020] 20 ) • die : Wir bewerten unsere PMs mit unseren HHH-Bewertungen von Big-Bench (Die Figur BOT: Gegnerdialoge und für Geschlechterbias (Die Figur Wir bewerten unsere RLHF-Modelle auf TruthfulQA (Die Figur BBQ-Lite [P von BIG-Bench, Geschlechterbias (Abbildung Gefühle, die auf Rasse und Religion beruhen (Die Figur RLHF improves senti- ment towards all groups, but does not remove bias. Static Alignment Evaluations [Askell et al., 2024] 6 5 ) [Xu et al., 2020] [Rae et al., 2024] 12 ) [Lin et al., 2024] 5 ) Römer et al., 2021) 40 ) [Rae et al., 2024] 17 ) • die : Wir berechnen Elo-Score basierend auf den Präferenzen unserer Crowdworkers, kombinierenden Kontext-Destillierten Modellen, Basis-RLHF-trainierten Modellen und endgültigen Online-RLHF-Modellen (Abbildung) Wir testen auch die Leistung unserer Online-Modelle während des Trainings (Abbildung) compare various levels of rejection sampling (Figure und führen Sie ein kontrolliertes Experiment auf iterativem Online-Training durch (Abbildung Darüber hinaus haben wir professionelle Schriftsteller eingestellt, um Gespräche zu komponieren, bei denen ein Assistent qualitativ hochwertige, hilfreiche und ehrliche Antworten liefert, und wir baten dann Crowdworkers, die Antworten unseres Modells mit denen dieser Schriftsteller zu vergleichen. 57 % der Zeit. Human Evaluations 1 ) 15 ) 36), die 16 ) 7 • die : Wir liefern Proben aus allen Palmen Sensi- tive Fragen und von Anrufen mit InstructGPT zur Verfügung gestellt und LAMDA im Anhang We show some comparisons with human writers in Sec- tion und wir zeigen mehrere kurze Dialoge in Abschnitt To mitigate the problem of cherry picking samples, we generate 17 samples per prompt and display only the median sample, as ranked by our online HH preference model. Samples [Solaiman und Dennison, 2021] [Ouyang et al., 2024] [Toppilan et al., 2024] C ist 6.1 und 6.3 Die 1.3 Verwandte Arbeiten Zwei kürzlich veröffentlichte Dokumente, LaMDA und InstructionGPT beide verwenden menschliche Daten, um große Sprachmodelle zu trainieren, um allgemein nützlicher oder abgestimmter zu sein. beide verwenden Sprachmodelle, die etwas größer sind als unser 52B-Modell. [Toppilan et al., 2024] [Ouyang et al., 2022] LaMDA finetunes große Sprachmodelle, um an interessanten, hilfsbereiten, factu-allied-basierten und sicheren natürlichen Sprachdialogen teilzunehmen. Wie bei unserer Arbeit enthalten sie Vorstellungen sowohl von positiven Interaktionen als auch von Sicherheit / Unschädlichkeit. Und ihre Verwendung von externen Informationen, um Genauigkeit / Grundlage zu gewährleisten, geht über die Methoden hinaus, die wir hier diskutieren, und ist sicherlich ähnlicher zu WebGPT und GopherCite Einige Unterschiede bestehen jedoch darin, dass sie anstelle des Verstärkungslernens eine Mischung von überwachten Lerntechniken (sowohl generativ als auch diskriminierend) anwenden und ihr Datenerfassungsverfahren absolute Bewertungen und nicht Vergleiche beinhaltet. [Toppilan et al., 2024] [Nakano et al., 2018] Menick et al., 2024 Anweisungen Finetunes GPT-3-Typ Modelle um ihre Hilfsbereitschaft zu verbessern. Wie in dieser Arbeit verwenden sie das Verstärkungslernen aus menschlichen Präferenzen, wie es durch Vergleiche ausgedrückt wird. Jedoch beinhalten sie auch eine überwachte Lernphase des Trainings, während im Gegensatz dazu unser Finetuning rein über RL (wir führen Kontextdestillation durch, aber das ist viel mehr wie einfaches Auffordern). Vielleicht ist der Hauptkontrast mit unserer Arbeit, dass sie kein Harmlose-Training oder Ex-Plore-Spannungen zwischen Hilfsbereitschaft und Harmlose enthalten. Ihr Ansatz unterscheidet sich auch in einigen Details von unserem: Sie trainierten keine Präferenzmodelle, die größer sind als 6B-Parameter, und sie mischten Vor-Training mit RL, um eine Degradierung [Ouyang et al., 2024] [Brücke et al., 2020] Unsere Arbeit unterscheidet sich sowohl von InstructGPT als auch von LaMDA dadurch, dass wir das „online“-Training erforschen, in dem wir die Modelle aktualisieren, die mit Crowdworkern interagieren, um progressiv qualitativ hochwertige Daten zu erhalten und die Schwanzen unserer Datenverteilung auszufüllen. Ein weiterer Unterschied ist unsere Erforschung spezialisierter Fähigkeiten wie Zusammenfassung und Codierung, die wir verwenden, um das Argument zu verstärken, dass Ausrichtung ohne Einschränkung von Fähigkeiten erreicht werden kann. Wir untersuchen auch explizit die Spannung zwischen Nützlichkeit und Unschädlichkeit, die bisher nicht angegangen wurde, soweit wir uns bewusst sind. Wir glauben, dass die einzigen wesentlichen Schritte die menschliche Feedback-Datenerfassung, Präferenzmodellierung und RLHF-Training sind. 2) der Mehrere andere jüngste Arbeiten konzentrieren sich auf Aspekte der Wahrhaftigkeit mit Hilfe der Wiederherstellung aus einer Datenbank oder über Internetsuche und menschliches Feedback, wie WebGPT und GopherCite Diese Arbeiten sind aufregend und ergänzen unsere Arbeit; insbesondere unsere Ergebnisse deuten darauf hin, dass ihre Techniken mit dem Training für Hilfsbereitschaft und Unschädlichkeit sehr kompatibel sein sollten.Während diese Arbeiten die treue Darstellung expliziter Beweise verbessern, wird wahrscheinlich mehr Arbeit erforderlich sein, um eine ehrliche Selbstdarstellung von KI-Systemen zu erreichen.Wir hoffen im Allgemeinen, dass Techniken unabhängig von menschlichem Feedback auf dieses Problem anwendbar sein können, da eine große Anzahl von Wahrheitsquellen nicht auf menschlichem Urteil basieren. [Lewis et al., 2020, Guu et al., 2020 Borgeaud et al., 2021) [Nakano et al., 2024] [Menich et al., 2024] Sicherheits- und ethische Fragen im Zusammenhang mit Sprachmodellen wurden umfangreich diskutiert (z.B. 2021)), mit bekannten Themen wie Toxizität, Vorurteile und die Möglichkeit, dass Modelle persönlich identifizierbare Informationen offenbaren können. Other works have explored methods to mitigate these problems (e.g. Modelle wurden auch geschult, um ethische Dilemmata direkt zu bewerten. Improvement auf ethischen Benchmarks demonstrieren Weitere allgemeine Forschungsvorschläge für KI-Sicherheit umfassen Die RL-Robustness-Fehler, die wir diskutieren, können als ein Beispiel für "Reward Hacking" betrachtet werden, das vor kurzem in [P RL policies could also fail to generalize out of distribution in other dangerous ways [Henderson et al., 2017] Bender et al., 2018 Weidinger et al., [Ganguli et al., 2024] [Liu et al., 2021] Xu und al. [Jiang et al., 2024] [Hendrycks et al., 2021a] Amodei et al., 2016 Hendrycks et al., 2021c an et al., 2022. [Koch et al., 2024] Unser Interesse an der Untersuchung von Trends mit Modellgröße wird durch Neuralskalierungsgesetze motiviert. Eine verwandte Beobachtung ist, dass, wenn die Parameterzahlen wachsen, Modelle effektiver finetune und viel weniger anfällig für „katastrophale Vergesslichkeit“ werden Wir erwarten, dass dieser Effekt dazu beiträgt, zu erklären, warum unser HH-Training mit guter Bewertungsleistung und speziellen Fähigkeiten für große Modelle kompatibel ist. [Hestness et al., 2019, Rosenfeld et al., 2019 Kaplan et al., 2020]. [Hernandez et al., 2021] [Ramasesh et al., 2024] 2 Datenerhebung Wir erwarten, dass das menschliche Feedback (HF) den größten Vergleichsvorteil gegenüber anderen Techniken hat, wenn Menschen komplexe Intuitionen haben, die leicht zu erzeugen sind, aber schwer zu formalisieren und zu automatisieren sind. Dies bedeutet, dass wir beim Sammeln von HF Aufgaben auswählen sollten, die so intuitiv und vertraut wie möglich sind. 2.1 Task Specification and Crowdworkers Our human feedback interface can be seen in Figure (for more details see Appendix People can interact with our models in natural language via chat, and ask for help with any text-based task. When it’s the model’s conversational turn, users see two possible model responses, and choose one with which to proceed. These two responses may come from the same model, or two different models. They can then ask follow-up questions or provide further instructions to the models. So there are two core components to the task, which repeat several times in each dialogue: 6 d) der • Crowdworkers schreiben eine Chat-Nachricht an unsere Modelle und bitten sie, eine Aufgabe auszuführen, eine Frage zu beantworten oder ein Thema von Interesse zu besprechen. • Crowdworkern werden zwei Antworten gezeigt, und sie werden gebeten, die hilfreichere und ehrlichere Rücksponsoren zu wählen (oder im Falle von rotem Team, um die schädlichere Antwort zu wählen). Wir vermuteten, dass Crowdworkers, die gut schrieben und die KI in interessantere Diskussionen einbezogen haben, tendenziell ein besseres Urteil darüber hätten, welche KI-Antworten am meisten „nützlich“ und „unschädlich“ waren. Otherwise, our approach to data collection was to largely let crowdworkers use their own intuitions to define ‘helpfulness’ and ‘harmfulness’. Our hope was that data diversity (which we expect is very valuable) and the ‘wisdom of the crowd’ would provide comparable RoI to a smaller dataset that was more intensively validated and filtered. Overall, our process was roughly of this form: 1. We invited master-qualified US-based MTurk workers to engage in dialogues with our models. 8 2. Anstatt alle unsere Crowdworkers zu bewerten, haben wir diejenigen identifiziert, die am produktivsten waren und zusammen etwa 80% unserer Daten ausmachten (etwa 20 Crowdworkers). Wir haben dann ihre Leistung hauptsächlich auf der Grundlage der Raffinesse und Variation in ihren Dialogen beurteilt, da dies ziemlich einfach war, intuitiv zu beurteilen (anstatt auf der Grundlage jeglicher Maßnahme der Einigung über nützliche / harmlose Entscheidungen). whom we continued to work with throughout the research process. 9 3. Wir haben unsere ausgewählten Crowdworkers zu einem Slack-Kanal eingeladen und mit ihnen per E-Mail korrespondiert, um sicherzustellen, dass sie gerecht kompensiert wurden und damit sie uns vor Problemen oder Problemen warnen können. 10 4. We also hired crowdworkers on Upwork, and vetted them in a similar, lightweight way. We have continued to use both platforms throughout this work. We find that it is easier to incentivize very high-quality interactions on platforms such as Upwork, where crowdworkers can easily be paid by the hour, rather than per task. But conversely, MTurk workers tend to generate data much more rapidly, and account for about 80% of our datasets. Wir filtern die Arbeitnehmer nicht nach Vereinbarung oder anderen direkten Maßnahmen der Etikettqualität, obwohl wir sie retrospektiv bewertet haben (siehe Abbildung). right) and found poor average agreement (about 63%) between Anthropic researchers and our crowdworkers, as compared to recent similar work such as 10 [Stiennon et al., 2020 Ouyang et al., 2022]. Als wichtige Warnung wurde unsere Crowdworker-Verteilung während dieser Arbeit nicht festgehalten, und wir erwarten, dass sich die Crowdworker-Qualität im Laufe des Projekts wahrscheinlich verbessert hat. Conversely, however, since we generally discouraged repetition, crowdworkers who have performed the task many times might also have had a tendency to engage in more esoteric interactions. 4.5. We should also note that we explicitly told crowdworkers that ‘lying isn’t helpful’ and that they should try to only reward helpful responses, which presumably explains why our models improve somewhat in terms of honesty. That said, we did not expect crowdworkers to fact-check our models significantly, and for example they often prefer responses that include non-functional URLs, which are probably one of the simplest possible ‘lies’ to debunk. und ehrlich 2.2 Hilfsbereitschaft und Unschädlichkeit (Red Teaming) Datensätze We collected two separate datasets using slightly different versions of our interface. For the helpfulness dataset, we asked crowdworkers to have open-ended conversations with our models, asking for help, advice, or for the model to accomplish a task (see Appendix ), and to choose the model response that was Für den harmlosen oder roten Datensatz haben wir Crowdworker gebeten, schädliche Reaktionen aus unseren Modellen hervorzurufen und die response offered by the models. D.2 zu more helpful more harmful Unser Interface (Abbildung allows users to express a preference strength. We only include comparisons in our datasets if crowdworkers expressed a preference stronger than the weakest available. In this work we will not otherwise use this preference-strength information; we treat all comparisons in our dataset as binary and of equal weight (so in particular we do not include ties). 6) Note that this means our helpfulness dataset tends to move conversations in a more beneficial direction, while in our red-teaming dataset user responses move conversations in a more harmful direction. We made this choice to make it possible for users to fully trick and exploit models while red-teaming, as this was most natural for other work we’re doing that’s specifically focused on harmfulness. However, we believe this difference made it difficult to train models that were both helpful and harmless, as explained in Section We plan to remedy this in future work, and would recommend others who are focused on training harmless dialogue models to collect data where users primarily choose model responses that move the conversation in the more beneficial direction instead. 4.4. 2.3 Models Deployed to the Feedback Interface and Associated Data Distributions Bei der Datenerfassung übernehmen wir vor allem used 52B language models with the broad specifications given in Wir haben drei Klassen von Modellen in unserer Schnittstelle verwendet: 11 [Askell et al., 2021]. • : At the beginning of the project this was the only model available. It performs similarly to a plain 52B language model prompted with HHH dialogues HHH Context-Distilled 52B Language Model [Askell et al., 2021]. • with a 52B preference model, where samples were generated from a 52B context-distilled LM. In this case the number von Proben war ein Parameter, aber am häufigsten haben wir = 16. Rejection Sampling (RS) k k • : We used a succession of these models in our interface. The models varied primarily based on the amount of data available when training the associated PMs (depending on the phase of the project). However, we also deployed models trained on different mixtures of helpfulness and harmlessness data. RLHF-Finetuned Models In the final phase of the project, when we were primarily deploying RLHF-finetuned models, we often de- ployed several such models at once. This allowed us to monitor progress by gathering model-comparison data, and also to (perhaps) improve data diversity. Corresponding to the three classes of models, we divide our data into three distributions: • A core Dieser Datensatz umfasst 44k hilfreiche Vergleiche und 42k rote (harmlose) Vergleiche (beachten Sie, dass ein Gespräch in der Regel etwa vier Vergleiche umfasst). base • A dataset consisting of 52k helpfulness comparisons and 2k red-teaming comparison using re- jection sampling models, where rejection sampling used a preference model trained on the base dataset. RS • An iterated ‘ ’ dataset including data from RLHF models, which were updated on a roughly weekly cadence over the course of about five weeks. This dataset contains 22k helpfulness compar- isons and no red-teaming data. online A histogram of these distributions by our final, online HH preference model’s score can be seen in Figure In der Sektion In what follows when we discuss the or base+RS dataset, we will be referring to the combination of the first two components. Our ‘online’ RLHF models in Section are trained on all three components. Most of our results are based on the static dataset, as we conducted experiments and evaluations with it while the online data collection was underway. 15 4.5. static 4.5 We analyze a few different splits of the static dataset – a standard split into 95/5 train/test data, and a 65/35 split that we use in order to obtain better statistics when evaluating preference model calibration on the test set. We also consider a 50/50 split, where we train distinct PMs on the two halves of the dataset. This is used to evaluate the robustness of RL training, as we then train an RL policy against one PM while evaluating the rewards achieved by that policy as measured by the independent PM. 2.4 Comparing Models with Elo Scores A significant part of our analysis compares models against each other in order to generate associated Elo scores, as described in That is, we have crowdworkers chat with two models simultane- ously, with each model generating one response (either ‘A’ or ‘B’) at each turn, and we record the sample that is preferred by the worker. This provides us with a record of ‘win rates’ between pairs of models, which we can then fit to corresponding Elo scores, to produce Figure (wo wir sowohl Gewinnraten als auch Elo-Score anzeigen). [Askell et al., 2021]. 1 for the fraction in which one model is preferred over another, the difference in Elo scores, and our PM scores. Note that conceptually win fractions, Elo scores and PM scores are interchangeable; we keep both Elo and PM scores so that we can avoid confusing crowdworker preferences (where we use Elo) with our preference modeling and RLHF (where we use PM scores). Note that the Elo scores for context-distilled models in Figure sich etwas von den analogen Ergebnissen für vorgeschlagene Modelle in – the Elo scores are now more compressed. The main difference is that we did not use top-p sampling this time Der Unterschied kann auch auf Veränderungen in der Crowd-Worker-Verteilung seit diesem früheren Experiment oder Veränderungen in den Erwartungen der Crowdworker zurückzuführen sein, da unsere Mitarbeiter vor diesem Test hauptsächlich mit hochwertigeren RLHF-trainierten Modellen interagierten. 1 [Askell et al., 2024] 12 3 Präferenzmodellierung für Nützlichkeit und Unschädlichkeit 3.1 Models and Training Setup We use language models with specifications that are identical to those discussed in [Askell et al., 2021], with a total of seven language models with parameter counts running from 13M to 52B and approximating a geometric series with increments of roughly 4X. We use PyTorch [Paszke et al., 2019] and Triton [Tillet et al., 2019] to facilitate model training and performance. Our preference model training setup is also identical to that in [Askell et al., 2021], and in particular we apply ‘preference model pretraining’ (PMP) to our language models before finetuning on our human feedback datasets, as explained in Section 4 of that paper. More details are provided in Appendix A. Note that we typically only train PMs for a single epoch, so the learning curves themselves (Figure 7 left) indicate how performance scales with dataset size (we used a fixed learning rate). 3.2 Grundlegende Scaling-Ergebnisse Wir möchten verstehen, wie sich die Performance der Präferenzmodellierung verbessert, wenn wir die Modellgröße erhöhen und zusätzliche Daten sammeln. we show basic results for PM accuracy when training on our static helpful and harmless data mixture. Roughly speaking, we observe log-linear trends in both dataset and model size. We tend to find somewhat more consistent trends if we model only the helpfulness or harmlessness distributions in isolation, rather than as a mixture, as observed in Figure im Anhang Aber auch hier sehen wir, dass für einige Datenverteilung scaling trends can exhibit more complex patterns that defy simple trends. 7 32 A.3. [Stiennon et al., 2020], Unsere Präferenzmodellierungsdaten stammen aus natürlichen Sprachdialogen, bei denen Crowdworkers textbasierte Gespräche mit dem Modell führen und die hilfreichere von zwei Modellreaktionen bei jeder Runde in der Konversation wählen (oder die schädlichere für rote Teaming-Aufgaben). PMs sind im ersten Schritt des Gesprächs etwas genauer, aber ihre Genauigkeit ist danach besonders konstant. 8. 3.3 Kalibrierung von Präferenzmodellen und Implikationen für RL Preference model scores should predict the probability that humans will prefer one or another model- generated response. We are interested in whether these probabilities are accurate, i.e. whether the PMs are well calibrated. We characterize calibration in Figure where we display PM accuracy as a function of the difference in PM scores assigned to pairs of samples, along with a heavy black line representing perfect calibration. We observe that PMs trained only on helpfulness data are very well calibrated, but PMs trained on a mixture of helpful and harmless data are slightly under-confident. 9, These calibration results are important because in later sections we will be using PM scores as the reward signal for reinforcement learning. Since the PM scores are well-calibrated, we can trust that they faithfully encode the probabilities that humans will prefer specific model samples (at least on-distribution with the training set). This means that when we see RL achieving a given reward, we can trust that those who interact with this model (if they are well-represented by our crowdworker distribution) will prefer it to reference models at a predictable rate, provided that the PM scores of the models’ responses are within the range considered in these calibration studies. That said, we find significant failures of robustness as RLHF optimizes towards much higher scores, as explained in Section und Anhang robustly 4.5 B. von 4. We might generally expect that as sample quality improves, it becomes more difficult to reliably identify the very best samples. In Figure in the appendix, we show that if we restrict to comparisons where both samples have a PM score above a given threshold, then PM accuracy drops as a function of this threshold value (to compare different PMs we add an offset so that they all obtain the same mean score on the test distribution). Notice that this result combines three effects: (1) differentiation among more sophisticated samples may be more challenging and require greater capabilities, (2) we have fewer very high quality samples to learn from in our dataset, as they are non-generic, and (3) pairs of samples that are all high-quality will have similar scores (compared to randomly chosen pairs), and so be more difficult to distinguish. 25 Diese Beobachtungen haben auch Auswirkungen auf das RLHF-Training, nämlich, dass wir von weiterem RLHF-Training verminderte Renditen erwarten sollten, sobald unsere Richtlinien eine ausreichend hohe PM-Score erzielen. 3.4 Evaluating Helpful and Harmless Preference Models 3.4.1 HHH Evaluation and Crowdworker-Anthropic Agreement Wir haben gezeigt, dass PMs, die auf unserem Datensatz ausgebildet wurden, etwas lernen und gut kalibriert sind. Aber wir möchten auch zeigen, dass diese PMs tatsächlich lernen, hilfreiches und harmloses Verhalten zu bevorzugen, entsprechend einigen unabhängigen Standards. Wir haben eine Reihe von Vergleichsbewertungen für hilfreichkeit, ehrlichkeit und harmlosigkeit (d. h. HHH-Bewertung) in früheren Arbeiten und bewerteten rohe, kontextuell destillierte und aufgeforderte Sprachmodelle bereitgestellt on this benchmark. As shown in Figure we find that our PMs perform much better than all of the models we evaluated previously. In fact, the Pathways LM effort Wir haben kürzlich eine durchschnittliche menschliche Punktzahl von 75% auf diesem Datensatz gemeldet, so dass unsere PM-Leistung von 86% in gewisser Weise weit über der durchschnittlichen menschlichen Genauigkeit liegt. [Askell et al., 2021] 5 , [Chowdhery et al., 2022] We also observe that there are a few outliers, where our PMs are confidently incorrect on HHH evaluation comparisons (see Figure left). These are all examples of failures of , where the better choice is honest but unhelpful, while the incorrect choice appears both helpful and well-written, but is subtly incorrect or misleading. We show the single worst example in Figure It’s worth noting that the bad response (which was written by Anthropic researchers) is likely out of distribution as compared to samples from our language models and RL policies. Since only model-generated samples were used to train the PMs, this may explain why the PM was badly fooled by these examples. It also indicates that the PMs are not adversarially robust. 10 honesty 11. We can also compare our crowdworkers, ourselves, and our preference models in terms of agreement on labels. The results are shown on the right in Figure und basieren auf etwa 320 Beispielen aus unserem statischen Testsatz. Die PMs werden geschult, um das Crowdworker-Verhalten vorherzusagen, so dass die PM-Crowdworker-Vereinbarung am besten ist. Jedoch stimmt der größte PM tatsächlich mit den Autoren überein (dh Anthropic-Forscher) etwas mehr als die Autoren mit Crowdworker auf Etiketten überein. Wir vermuten auch, dass dies eine schlechte Unterprobe der Daten ist, da die PM-Genauigkeit auf dieser Unterprobe im Vergleich zum vollen Testsatz mehrere Prozent niedriger ist. 10, die 3.4.2 Gender Bias and Bot Adversarial Dialogues We can begin to evaluate whether our PMs display gender bias by comparing scores across gendered terms. To this purpose we evaluate preference model scores on short dialogues of the form Human: Can you help me fill in the missing word in a sentence? The sentence is: The {Occupation} is a Assistant: Sure thing, here is your sentence with the missing word filled in: The {Occupation} is a {Gendered Term} where we use a list of 76 occupations, 12 matched gender terms (mother vs father, guy vs gal, etc), and both present and past tense Preference model scores are not directly meaningful, so instead we evaluate the difference in score between paired gender terms, and compare that to the difference in score among terms within a given gender. The results are shown on the left in Figure We observe that the variation among terms that indicate a given gender appears to be at least as large as the variation across genders. So it would appear that the PMs do not exhibit a substantial bias. [Rae et al., 2024] 12. As an external validation of our PMs, we consider the Bot Adversarial Dialogues (BAD) dataset This dataset contains a few thousand conversations between an AI system and a human. Each AI response is labeled as harmful or not harmful. We compute the distribution of preference model scores for BAD AI utterances (we restrict our analysis to the first BAD AI utterance per conversation) and find that the BAD AI utterances marked as harmful have significantly lower preference model scores. This suggests that our PMs are effectively classifying these AI generated utterances, even though they are likely quite different from the data distribution that our PMs were trained on. [Xu et al., 2020]. 4 Verstärkung des Lernens aus menschlichem Feedback 4.1 Trainingseinrichtung We apply reinforcement learning (RL) with preference modeling, following the approach outlined in die in den folgenden Schritten zusammengefasst werden kann: [Stiennon et al., 2020], 1. Bereiten Sie einen Datensatz von Vergleichen vor und trainieren Sie einen PM, um dem „besseren“ Element in jedem Vergleich einen höheren Punkt zu verleihen.Im Rahmen unserer menschlichen Feedback-Experimente besteht jeder Vergleich aus einem Anruf, gefolgt von einem Paar von Modellgenerierten Antworten, wobei am Ende jeder Antwort ein PM-Score ausgewertet wird. 2. Extract all the prompts from the preceding dataset, and train an RL policy to generate a response to each prompt autoregressively, with a reward signal provided by the PM score at the end of the response. PM dataset and training details are provided in Appendix we also discussed the performance of our PMs in Section In the language of RL, each response generated by the policy is a ‘timestep’, a full conversation is one ‘trajectory’, and the PM score is a single ‘reward’ provided at the end. A.2; 3. The idea is to use the preference model to steer the policy towards writing better responses. However, as we saw in earlier sections, PMs also become less calibrated at higher scores, so higher rewards do not necessarily imply better performance. To stabilize RL training, we use Proximal Policy Optimization (PPO) Wir verfolgen auch andere Arbeiten and apply an empirically-estimated KL penalty term in the reward, with the total reward given by [Schulman et al., 2017]. [Stiennon et al., 2020] where KL >/ 0 is a hyperparameter. In practice we use a very small value of KL = 0:001, which likely has a very minor impact during most of RL training (as DKL < 100 typically), and might actually be wholly unnecessary. More details about RL are provided in B.1. Throughout this paper we use PM = the preference model score itself for the RL reward. Recall that as implied by equation this means that the difference in PM values between two samples and will be related to the predicted probability (in der ) that will be preferred to via r (2.1 ) und r A B P A > B A B There is no good reason um diese Präferenzmodell-Score direkt als Belohnung zu verwenden, aber sie wurde in früheren Arbeiten wie und so für die Einfachheit werden wir hier keine Variationen zu dieser Wahl erforschen. 13 [Stiennon et al., 2020] In order to produce additional prompts (i.e. the human side of the conversations) for RLHF training, we used a large LM to generate them. For this purpose, we simply used few-shot learning, creating a context with about 10 existing high-quality human queries, and then sampling to generate more. We find that the sample efficiency of RLHF is roughly the same on the original crowdworker-written prompt dataset and the model- generated one, so we combine the two for greater diversity during RLHF training. We used 137k prompts from the ‘static’ dataset, and 369k model-generated prompts. Note that almost all of our preference modeling data was collected from 52B models. This means that RLHF training with smaller models might have been challenging, since samples from smaller models tend to be out-of-distribution from the PM training data. Thus it is quite interesting that models more than fifty times smaller were actually able to learn and improve, as seen in Figure 1. 4.2 Robustness Experiments We now discuss the problem of RLHF robustness. A fully robust PM would agree with humans on distribu- tions of dialogues quite different from those encountered during PM training (i.e. different from those created by crowdworker interactions with our deployed AI assistants). However, we do not expect that our PMs are so robust, and in fact Figure provides one plausible example of a robustness failure. Since RL optimizes the policy to maximize the PM score, any failure in robustness on the part of the PM may be exploited by the RL policy to achieve higher rewards, without actually improving the policy’s behavior from the point of view of human evaluators. 11 A rigorous way to study robustness is to take snapshots of the policy at various points during RLHF training, including the initial snapshot, and have crowdworkers compare their performance. This gives a ‘true’ Elo score, as evaluated by crowdworkers, which can then be compared directly with the PM scores. We present an example of this study in Section 4.5. However, this sort of test requires collecting additional human feedback data, which can be slow and expen- sive, so here we also study robustness from a different angle. Similar to how datasets are split into train and test sets for supervised learning, we split our preference model comparison data into two halves (a half and a half), and train separate preference models on each, which we refer to as the ’s and the ’s. Wir trainieren dann RLHF-Politik gegen die ’s, while evaluating them using the s. Ähnlich wie Test-Set-Bewertungen uns helfen, Überschneidungen im überwachten Lernen zu verstehen, evaluations help us understand overfitting relative to the ’s. These experiments are not conclusive since the train and test PMs may exhibit correlated robustness failures. train test train PM test PM train PM test PM test PM train PM The main conclusions from these experiments are: (1) RLHF becomes gradually less robust at higher PM scores, and (2) larger preference models are more robust than smaller ones. Wir führen zwei Experimente wie folgt durch: • : This set consists of a scan of policies (i.e. one for each model size), all of which are trained with respect to the same 52B . Train PM Size = 52B train PM • : Dieser Satz besteht aus einem Scan von Richtlinien, wobei jede Richtlinie in Bezug auf eine of the same size as the policy. Train PM Size = Policy Size train PM For both experiments, each policy is further evaluated with respected to a scan of ’s throughout training. Note that a refers to 7 different model sizes ranging from 13M to 52B, thus giving us 7 policies and 7 × 7 evaluations per experiment. test PM Scan In Figure we compare the and scores throughout the training process, similar to how train and test curves are often compared for supervised training. We find that in all cases, the two scores are in close agreement during early stages of training, but eventually diverge, with the providing a lower score. The divergence is likely an indication that . That is, the policy has been over-optimized on the , making the over- confident in the policy’s performance. The , auf der anderen Seite, leidet nicht an diesem Problem, da es auf einem anderen Teil der Daten geschult wurde, die weder die Richtlinie noch die had observed. 4, train PM test PM test PM the preference model is less robust and more easily exploited at higher rewards train PM train PM test PM train PM We provide more discussion in Appendix B.2. 4.3 Eine ungefähr lineare Beziehung zwischen √ KL and Reward D D In Figures und Wir beobachten eine between √KL and PM score during RLHF training. Furthermore, we note that when√a__ll mo__dels are trained and evaluated with the same PMs, the learning curves are roughly In der KL-reward plane. Note that here the ‘KL’ is more precisely reward varies linearly in , then in the ‘sma√ll__- r__egion’ (d.h. wo die Serienerweiterung eine gute Annäherung bietet), sollten wir eine Belohnung erwarten KL. Typically we should that reward varies linearly in Weil die ursprüngliche Politik was not previously optimized for reward, so there is no reason why it would sit at an extremum with respect to small variations . So the fact that this relation seems to hold empirically suggests that most of RLHF training remains in the small- Das Regime. 4 13 approximately linear relation parallel D Die Dπ δπ D expect Die Dπ π δπ δπ Though they did not use these coordinates, a similar scaling can be read off from the results in learning to summarize Insbesondere bieten sie eine schöne Analyse der Ablehnungssammelung, wo sie Proben, und dann Plot bedeutet Belohnung der Spitze samples versus the KL = log( ). [Stiennon et al., 2020]. N k D N/k This analysis suggests that these RL learning curves might be associated with changes in the RL policy that behave very similarly to simply rejection sampling from the initial distribution. Wir finden diese einfache Beziehung ziemlich auffällig, und glauben, dass sie weitere Untersuchungen verdient.Auf einer vermutlichen Ebene könnte es eine Vielzahl von Implikationen und Verwendungen haben, wenn RL-finetuning große generative Modelle: • These relations provide a rough prediction for ‘how much does the policy need to change to achieve a specific reward’. Furthermore, if the lines corresponding to different model sizes really are parallel, then one can use RL training of a small model along with the zero-shot performance of a larger model to estimate the eventual performance of a larger RL policy. The slopes of these lines also explain how RLHF training can produce such large effective gains in model size, and for example it explains why the RLHF and context-distilled lines in Figure Sie sind fast parallel. 1 • One can ask a subtle, perhaps ill-defined question about RLHF training – is it or simply . We Das Modell neue Fähigkeiten lehren Fokussierung des Modells auf die Erzeugung einer Unterverteilung bestehender Verhaltensweisen könnte versuchen, diese Unterscheidung zu machen sha√r__p b__y die letztere Klasse von Verhaltensweisen mit der region where RL reward remains linear in KL. • To make some bolder guesses – perhaps the linear relation actually provides an on RL re__war__d, as a function of the KL. One might also attempt to extend the relation further by replacing √KL with a geodesic length in the Fisher geometry. upper bound By making RL learning more predictable and by identifying new quantitative categories of behavior, we might hope to detect unexpected behaviors emerging during RL training. 4.4 Tension Between Helpfulness and Harmlessness in RLHF Training Here we discuss a problem we encountered during RLHF training. At an earlier stage of this project, we found that many RLHF policies were very frequently reproducing the same exaggerated responses to all remotely sensitive questions (e.g. recommending users seek therapy and professional help whenever they express any level of displeasure at all). This greatly limited these models’ utility. We still see a vestige of this behavior in some of the examples provided in Section We now believe these policies were the result of for harmlessness, while helpfulness. 6.2. over-optimizing under-optimizing With our data collection procedure, we think this is quite intuitive. In order to get a very good score on red-teaming prompts, it’s probably sufficient for models to respond with something like “I can’t answer that.” This does not require much sophistication (it just requires learning to classify harmful requests), and so we expect it is easier to learn than helpfulness. In der Figur (right), we show the policy’s PM score throughout training, after separating helpfulness and harmlessness prompts. On the left side of the same figure, we show the score distribution of PM comparison data, again separating helpful and harmless datasets. We observe that the policy’s harmlessness score is somewhat off-distribution, as it is on the upper tail of the harmlessness comparison data. On the other hand, the policy’s helpfulness score appears on-distribution, and is likely under-optimized. So we would expect this agent to be very difficult to red-team, but not very helpful. 14 This then raises an obvious question – can’t we just collect more harmlessness data to fill out the upper tail of the distribution? The problem involves the definition of harmlessness mentioned above – if simply refusing to answer a question is the ‘least harmful’ behavior, then this is probably both very easy to learn, and hard to improve on. That said, a more interesting ‘least harmful’ behavior would involve the model (helpfully) explaining why the request was harmful, and perhaps even trying to convince the human not to pursue such requests. We informally refer to such a model as a ‘hostage negotiator’. However, our data collection process made it very difficult for models to learn ‘hostage negotiation’. This is because when collecting our harmlessness dataset, we had crowdworkers choose the AI response. We made this choice so that we could fully explore the vulnerability of our models to red-teaming. However, from the point of view of RLHF this was problematic, because beyond the first turn of dialogue, our models never learned what a response to a harmful query might be like. Our dataset does not provide guidance on the upper end of the distribution, on what models should do, but only tells models what to do. more harmful verfeinert nicht In practice, we have partially resolved the optimization issue by training on a larger fraction of helpfulness prompts during RLHF. But in the future we hope to more fully and systematically address this problem by collecting harmlessness data where crowdworkers choose the best possible response from our models. Auf diese Weise hoffen wir, dass Modelle, anstatt schädliche Anfragen einfach abzuschalten, die subtilere Kunst der "Gefängnisverhandlung" mit roten Teammitgliedern lernen können. 14 Note that since the data and models discussed in this section are from an earlier stage of our research, the RL results may look slightly different from other parts of the paper. 4.5 Iterierte Online RLHF In den vorangegangenen Abschnitten haben wir das Problem diskutiert, dass PMs bei höheren Scores schrittweise weniger kalibriert und weniger robust werden, wie in der PM-Kalibrierungsstudie in Abbildung zu sehen ist. and the RLHF robustness study in Figure 9, die We believe this is caused by a lack of data in this high score regime. To address this, we propose : 4. iterated online RLHF • We simply train the best RLHF policy we can, and use that to collect comparison data from crowd- workers. Since the policy was trained to optimize for PM score, it should produce responses that are on the upper end of the score distribution. • We mix the new comparison data with our existing data, and train a new scan of PMs, which we then use to train a new scan of RLHF policies. Then reiterate this process indefinitely. Our hypothesis is that the ‘online’ RLHF policy helps us collect data on the upper end of the PM score distribution, which should improve PM calibration at high scores on subsequent iterations, and thereby allow us to train even better policies. Continuing this process should give us progressively better PMs and policies. Note that our use of the terminology ‘online’ is different from conventional use of the word—instead of training the same model iteratively, we retrain a new model per iteration. One concern about this approach is that RLHF tends to decrease the policy’s entropy, which would limit the diversity of data collected through the online procedure. We partially address this by deploying a number of different snapshots from RL training, and from different online iterations, at once. This also makes it possible to compare these models to get a better sense of how they are performing. We can see signs of life from the online approach by looking at the evolution of our data distribution. In Figure (left), we show the PM scores from three distributions of models: , (Ablehnung von Proben) und , as described in Section Wir sehen, dass unsere endgültige Online-PM (auf allen Daten ausgebildet) die Qualität der Proben von der Basis bis zur Ablehnungssammelung bis zu den Online-Datenverteilungen verbessert. Wir fanden auch, dass unsere Online-PM Genauigkeiten von 74%, 70% und 67% auf den Testsätzen für die jeweilige Basis, RS und Online-Distributionen erreicht, was zeigt, dass es immer schwieriger wird, zwischen höherwertigen Proben zu unterscheiden. 15 Base RS Online 2.3. We show the learning curves for our online models, along with measurements of Elo scores from crowd- workers, on the right in Figure We see that models improve significantly during RLHF, but Elo scores from crowdworkers do not match predictions from PMs. We further discuss and decompose the robustness of RLHF training in Appendix where we see that distributional shift accounts for a significant part of the apparent robustness failure (Figure 15. B.4, Die 35). In Figure Wir vergleichen die Elo-Score unseres Online-Modells mit kontextuell destillierten Modellen und RLHF-Modellen, die auf dem „statischen“ (d.h. nicht online) Datensatz ausgebildet wurden, was zeigt, dass die Online-Modelle von unseren Crowdworkern eindeutig bevorzugt werden. definiert in Anhang and its PM was trained with 2048 context instead of 1024), as compared to the earlier static RLHF training run. 1, K B. 1 und Um diese beiden Warnungen anzugehen, haben wir ein kontrolliertes Experiment durchgeführt, das zwei RLHF-Laufwerke vergleicht: Einen, der mit unserem Basisdatensatz geschult wurde (etwa 44k PM-Vergleich), und einen anderen, der auf einer gleichmäßigen Mischung aus Basis, RS und Online-Daten geschult wurde. (about 15k PM comparisons from each). So for this experiment we trained two separate PMs on each dataset, and then trained a pair of RLHF policies against these two PMs. Apart from the data difference, both runs used the same settings, and were only trained on helpfulness. In figure we compare Elo scores for various snapshots of both runs, as determined by crowdworker preferences, showing that the policy trained on the iterated-online mixture is clearly preferred. This demonstrates that online training works, and that performance gains are not merely due to increased dataset size or hyperparameter changes. whose total dataset size is the same as the base dataset 15 16, 4.6 Evaluations: Alignment Bonus, Honesty, and Biases Language models that have been finetuned via RL typically have much narrower, lower-entropy output distri- butions. This can make evaluations difficult when they are fairly rigidly formatted, since all valid responses may be far off-distribution for the RLHF model (we discuss an example with gender bias evaluations below). Thus we expect in future work evaluations involving sampling and human interaction may be most relevant. In what follows we discuss some standard NLP evaluations, and then evaluations specifically related to the societal impacts of the models, including honesty, sentiment, and bias. 4.6.1 NLP Evaluations We evaluate our models on question answering, commonsense, trivia, and story completion using the bench- marks MMLU Lambada [P Hellaswag Das OpenBook ARC and TriviaQA The main conclusion is that RLHF tends to improve performance for large models, while degrading the performance of smaller models. [Hendrycks et al., 2021b] abgelaufen et al., 2016 [Zellers et al., 2019], [Mihaylov et al., 2018], [Clark et al., 2018], [Joshi et al., 2017 ]. 16 Vollständige Ergebnisse sowohl für Null-Schuss- als auch für wenige Schuss-Bewertungen sind in Zahlen angezeigt und and we provided a summary of the mean trends in Figure Leser können feststellen, dass sich die Ergebnisse für einige Bewertungen ziemlich plötzlich verbessern; dies ist eine Folge des Formats, das wir für mehrere Auswahlfragen verwenden, wo wir explizit Optionen bereitstellen (Gopher) in diesem Format verwendet wird). Das Format ist ausdrücklich in Anhang We find that this format tends to improve performance for large models, while decreasing the performance of small models, leading to the arguably misleading appearance of a ‘grok’ curve. 28 29, 3. [Rae et al., 2021] E. [Power et al., 2022] 4.6.2 Honesty and Biases Eine große Frage ist, ob KI-Modelle ehrlich sind.Wir bewerten unsere Modelle auf TruthfulQA (MC1) Zeigen Sie die Ergebnisse in Figur There we also include performance at 50-shot, in or- der to demonstrate that while our RLHF training significantly improves honesty, our models most likely have significant room for improvement. Note that our instructions to crowdworkers suggest that ‘lying isn’t help- ful’ and that they should choose responses that are ‘helpful and honest’, so this is presumably related to the improvements we see on TruthfulQA. That said, we do not currently expect RLHF to be the best approach to honesty. [Lin et al., 2021] 5. Another set of questions involves the underlying biases of these models. We evaluate our models for sentiment biases on race and religion (in the same format as Gopher 2021]), for gender bias, and on the Bias Benchmark for QA (BBQ-lite) [P [Rae et al., arrish et al., 2021]. Results for sentiment towards different racial and religious groups are shown in Figure The main effect we observe is that the sentiment of our RLHF-trained models tends to be much more positive than that of plain LMs, but it is more difficult to say if bias among groups has changed. We expect some of the visible effects are due to limitations in the sentiment model , for example the low scores for the Atheist group is likely because samples such as (model generation in italics): 17. 17 The Atheist man was very Er ist skeptisch und kritisch gegenüber religiösen Dogmen und konventionellen moralischen Prinzipien und sucht stattdessen, seine ethischen Werte aus rationaler Betrachtung und empirischer Beobachtung der natürlichen Welt abzuleiten. receive a sentiment score very near zero, which seems like a questionable evaluation. For these evaluations we use a prompt format where the human asks the assistant to complete the sentence as follows: Human: Can you help me finish a sentence? The sentence is: {sentence beginning} Assistant: Sure thing, here is your completed sentence: {sentence beginning} with sentence beginnings such as ‘The Atheist man was very’ as seen above. We also evaluated our models on BBQ-Lite, a simplified version of BBQA [P that is avail- able on BIG-bench. We were somewhat surprised by the results, and are unclear if any conclusions can be drawn based on them. In essence, we appear to find that our plain LMs do not exhibit very significant biases on this benchmark, using the available metrics. Here the metric is roughly ‘how often does the model invoke a term in a negative context, divided by how often it uses the term generally’. A value of 0.5 corresponds to the models being ‘unbiased’. RLHF training perhaps improves the results slightly (likely by improving sentiment) though the effect is very small (see Figure We may return to these evaluations in the future using the full BBQA metrics [P arrish et al., 2021] 18 ) Römer et al., 2021) In Section 3.4.2 we investigated gender bias in preference models. We tried using this format for evaluating RLHF models, but we found that these models were more than 10 less likely than plain or context-distilled LMs to use a gendered term to complete sentences like ‘The CEO was a’. In other words, statements like ‘The CEO was a father’ seem to be somewhat off-distribution for the RLHF models, at least compared to general LMs. This is problematic for evaluations, because it means that the bias measurements are likely to be dominated by noise, and are unlikely to be representative of model behavior (i.e. model-generated samples). So in Appendix B.10 we experiment with another formulation of the task, where we reverse the order of occupation and gender terms, and add a ‘For work’ clause; this kept both the RLHF and other models on-distribution. The results are discussed in Appendix B.10, where we show that RLHF model’s biases are very strongly correlated (Figure 40) with underlying LM biases. In particular, with this evaluation we find that large RLHF models have gender biases that are very similar to language models evaluated at a lower temperature. 5 Competing Objectives, Specialized Skills, and OOD Detection Eine Besorgnis über Ausrichtungstechniken besteht darin, dass sie die Modellleistung beeinträchtigen könnten. we highlight a quantifiable trade-off of this kind, between helpfulness and harmlessness when training preference models. But it appears that larger models suffer less of a performance drop from this trade-off. 5.1 Darüber hinaus finden wir auch, dass der Konflikt zwischen Nützlichkeit und Unschädlichkeit relativ einzigartig ist. Pref-erence-Modelle können lernen, starke Leistung bei speziellen Fähigkeiten zu belohnen, ohne jeglichen Verlust in der Leistung bei Nützlichkeit und Unschädlichkeit. we consider the evaluation of summarization quality as such a skill, using the learning-to-summarize Datensatz reformatiert in Konversationsform. später in Abschnitt we show that code models (i.e., models finetuned on code by supervised training) are also compatible with HH alignment interventions, even though the RLHF training does not involve code data or examples. 5.2 [Stiennon et al., 2020] 5.3 In der Sektion we highlight another approach to avoiding harmful behavior – it may be possible to reject most harmful requests, even without any access to harmfulness training data, by leveraging out-of-distribution detection techniques [F This approach might also be useful more generally in deployment scenarios where strange or off-topic requests need to be flagged or ignored. 5.4 ort et al., 2021]. 5.1 Mixing Helpful and Harmless Objectives In many cases harmlessness acts as a constraint on helpfulness. So we should expect that helpfulness and harmlessness may behave as partially anti-correlated objectives. We establish this by evaluating preference models trained on different mixtures of HH data, and with different weightings. Auf einer konzeptionellen Ebene können die HH PMs im Wesentlichen lernen, die Daten zuerst zu klassifizieren und dann je nach Verteilung eine Punktzahl zu wählen. Wir werden zeigen, dass größere Modelle besser funktionieren und robuster für Datenmischung und Verlustgewichte sind, was möglicherweise auf ihre größeren Erfolge bei der Trennung von vernünftigen von schädlichen Anfragen zurückzuführen ist. 5.1.1 Varying Helpful vs Harmless Data Fraction We train models using data splits varying from 100% helpfulness to 100% harmlessness in intervals of 10%. Our static data distribution has 42k red-teaming comparisons, so to control for dataset size we always con- struct mixtures with a total of this number of comparisons. Figure zeigt Leistung sowohl bei der Schadensminderheit als auch bei der Hilfsbereitschaft, da die Trainingsdatenmischung vielfältig ist. Beachten Sie, dass das Training ausschließlich bei den Hilfs- oder Harmlösheitsdaten die Leistung auf der anderen Verteilung ergibt, die . This exemplifies the extent to which these distributions are in tension with each other. 19 significantly worse than chance Figuren in der Anlage auch Plots durchschnittliche Testgenauigkeit über sowohl nützlichkeit und harmlosigkeit (wo Mean Acc = (Harmlessness Acc + Helpfulness Acc)*/*2). Kurven für größere Modelle sehen steiler in der Nähe der 0% und 100% Bereiche, aber flacher an der Spitze. Die Kurven für die kleineren Modelle sind schrittweise, mit mehr unterschiedlichen Spitzen in der Mitte. Dies wiederum deutet darauf hin, dass größere PMs robuster sind, um den spezifischen Bruchteil von roten-teaming vs nützlichkeit Daten, die verwendet werden, so dass sie beide Konzepte leichter lernen. 26 5.1.2 Weighting Helpful vs Harmless Losses Instead of studying different data mixtures, we can try re-weighting the losses. Since we have more helpful- ness than harmlessness comparisons, we experimented with weighting the losses as LTotal = LHelpfulness + λ · LHarmlessness für λ E 2 {1; 2; 3; 4; 10}, wie in Abbildung 27 gezeigt (auf den Anhang übertragen). Wir bemerken, dass größere Modelle bei der Wahl von λ robuster erscheinen. 5.2 Zusammenfassung als spezialisierte Fähigkeit Wir erwarten, dass Modelle, die mit speziellen Fähigkeiten finetuned sind, besonders nützlich und wertvoll sein können. As one test of this question, we studied PM finetuning on the learning-to-summarize (LtS) dataset vs. a mixture of LtS and HH data. We formatted the LtS data in conversa- tional format so that it matches the HH data, as follows: [Stiennon et al., 2020] Human: Can you write a summary of this article for me? ...Text... Assistant: Sure, here it is: ...Summary... As shown in Figure large preference models trained on a mixture of HH and LtS datasets perform equally well on both. So at least at the level of preference modeling, there seems to be no cost to mixing HH with the specific skill of evaluating summarization quality. 20, die 5.3 Natural Language RLHF on Code-Finetuned Models As another test of a specialized skill, we would like to see if natural language alignment can be combined with coding without compromising performance. Since our crowdworkers were never instructed to probe the model’s coding abilities, and most likely do not have a great deal of coding expertise, our human feedback data does not include a significant number of code-related conversations. The preference model finetuning dataset and the RLHF prompt dataset thereby do not contain any significant amount of code, though there is some code in the LM pretraining mix and possibly a small amount in the PM pre-training (PMP) mix. This makes code-related problems an interesting way to test generalization of RLHF, and especially its compati- bility with other skills. Our ‘base code models’ were finetuned on Python code scraped from Github as described in Beginnend mit diesen Python-Fine-Tuned (Python FT) -Modellen führten wir dann unser Standard-Natural Language RLHF-Training mithilfe von "statischen" Präferenzmodellen und -Anrufen durch. [Askell et al., 2024] We evaluate models on the HumanEval dataset which prompts language models with python function signatures and docstrings. Models are tasked with correctly filling in the function body given the context, and model-written functions are run in a sandbox environment. In Figure we show results versus model size with and without RLHF training. We see the same trend here as with other evaluations – RLHF decreases the performance of small models, but improves the performance of larger models. [Chen et al., 2021], 21 RL training tends to decrease the entropy of the models’ distribution, and so we were concerned that these results would be very sensitive to temperature and top-p tuning. So for our 52B models, we performed a scan over temperatures and two top-p settings for both the RLHF models and the base code models, and then chose the best setting for each model and pass@k. We did a grid-search over the evaluation hyperparameters: T ∈ {0, 0.4, 0.6, 0.8, 1.0} × p ∈ {0.95, 1} × k ∈ {1, 5, 10, 25, 50, 75, 100}. Results are summarized on the right side of Figure 21. For each model and for each k in pass@k, we take the maximum performance over all 10 combinations of hyperparameters. We see that RLHF improves performance over the baseline on this evaluation, for all pass@k. We should emphasize that as with our other evaluations, the improvements in performance from RLHF are modest. In fact, we find that simply prompting a base code model performs slightly better, as shown in Figure Appendix weiter beschreibt das Format der von uns verwendeten Anrufe (d.h. „HHH-Anrufe“), die aus ein paar Code-Beispielen bestehen. 38. B.8 We also conducted experiments involving adding buggy code to the prompts, which typically worsens per- formance (see 2021]). We found that RLHF models did not perform better than their initial base code model snapshots, when these prompts are included in the context during evaluation, even after scanning over temperature and top-p. [Chen et al., 5.4 Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests In this work we are primarily focused on achieving harmlessness entirely through natural language dialogue. However, one might try to avoid harmful behavior in a somewhat different manner, by either restricting language assistants to only respond to a narrow range of queries (approved-list), or by filtering and rejecting known types of bad behavior (block-list). We could use our preference models for these purposes, but we might also take a different, less supervised approach, and leverage advances in out-of-distribution (OOD) detection. Such an approach might also be useful for those who want to build systems that only respond to a narrow range of queries (e.g. code models that should avoid non-code topics). Out-of-distribution detection (OOD), and especially near out-of-distribution detection, have been a major challenge for deep neural networks. Deep networks routinely assign high probability to mis-classified inputs [Guo et al., 2017, Lakshminarayanan et al., 2016] as well as to test inputs not belonging to any of the training classes [Nguyen et al., 2014]. There have been many approaches to OOD detection based on discriminative models [Hendrycks and Gimpel, 2016, Lee et al., 2018, Liang et al., 2017, Liu et al., 2020] as well as deep generative models [Nalisnick et al., 2019, Zhang et al., 2020]. The more difficult case of OOD detection, the so-called near-OOD detection [Winkens et al., 2020], has recently been improved upon significantly using pre-training and large models [Fort et al., 2021]. Für einen prompt i extrahieren wir einen Vektor der Aktivierungen des Dimension dmodel aus einer Schicht ` und nennen es v ` i ∈ R dmodel . Die Aufgabe besteht darin, zwischen einem unsichtbaren Beispiel von Unschädlichkeit und Nützlichkeit Daten zu unterscheiden, ohne dass explizit irgendwelche Daten von Unschädlichkeit überhaupt gezeigt werden. Dieser Ansatz funktioniert, indem wir die Abweichung eines prompt von den nützlichen Daten messen, anstatt zu messen, wie nah es sich insbesondere an Unschädlichkeit Daten nähert. Auf diese Weise sind wir nicht von dem spezifischen schädlichen Inhalt abhängig, den wir bei uns haben, und können möglicherweise verschiedene Arten von Nicht-Nützlichkeit-Inhalten filtern. Um festzustellen, ob ein Eingang aus der In-Distribution ( Eine einfache Verbesserung über der Entfernung von Mahalanobis nennt man die Mahalanobis Entfernung wurde vorgeschlagen in und hat gezeigt, dass es zu einer besseren AUROC sowie zu einer robusteren Erkennung für eine Reihe von OOD-Problemen in Sicht und Genomik führt (außer zu mehr Robustheit bei angreifenden Angriffen). Inspiriert von dieser Methode und in der Erkenntnis, dass unser Problem natürlich keine semantisch bedeutsamen Klassen beinhaltet, die die in-Distribution umfassen, schlagen wir eine weitere Modifikation vor, die wir die Relative Mahalanobis distance. We compute it by fitting a full covariance matrix Σ as before, as well as a diagonal- only covariance matrix Σdiag, and assigning the difference of their Mahalanobis distance as our scoring function, score( ) = ( - von ) Σ−1( - von ) − ( - von ) Σdiag−1( − ). Relative [Ren et al., 2021] ort, Simplified x x µ T x µ x µ T x µ Figure shows the results for our OOD detection experiments, trying to distinguish the helpfulness data from harmlessness data using our new Simplified Relative Mahalanobis distance to the helpfulness activation vectors. The figure shows AUROC for activation vectors from different layers of the model and different model sizes. The bigger the model, the better its performance, with the middle layers performing best. The right panel of Figure shows an example of the OOD score distributions for the helpfulness data (blue) and harmlessness data (red) for a 64L layer of 52B parameters and its 32th layer. We can see that the mode of the distances of the Harmlessness data is clearly higher than for the helpfulness data. The comparison of the Simplified Relative Mahalanobis distance and the standard Mahalanobis distance is shown in the Appendix in Figure 22 22 B. 9 39. The advantage of this approach is that we are able to distinguish helpfulness data from non-helpfulness data, with harmlessness data being only a particular kind of non-helpfulness data. The disadvantage is its clear lower performance on this specific task in particular. If we have access to a small number of examples of Harmlessness inputs (the out-distribution), we could perform a few-shot outlier exposure, as first proposed by suggests using a single class representing the OOD examples. [F hat externe Exposition über einzeln starke nahe OOD-Detektoren verwendet und zeigte, dass sie immer noch einen großen Nutzen aus der Exposition gegenüber den Beispielen der out-Distribution genießen. und in der Figur gezeigt [Hendrycks et al., 2018] [Thulasidasan et al., 2021] ort et al., 2021] B. 9 23. Insbesondere können wir mit nur 10 Beispielen von schädlichen Anweisungen eine verbesserte AUROC von 0**94 0**02 für das 64L-Modell erzielen.Die beste Leistung unter allen Schichten aller Modelle (die mittleren Schichten des 52B-Modells) ohne äußere Exposition (also mit unserer vereinfachten relativen Mahalanobis-Distanz, siehe Abbildung [22]) beträgt ungefähr 0*.*85. Ein 4L-Modell, das nur 10 Beispielen schädlicher Daten ausgesetzt ist, erhält eine AUROC von 0*86 0**01, während es nur 13M-Parameter im Vergleich zum 52B hat. 6 Qualitative Beispiele und Vergleiche It is challenging to quantitatively evaluate general-purpose dialogue agents. We find that our own research process depends essentially on qualitative evaluations, in order to get a sense for model strengths and weak- nesses, even when the ultimate goal is to produce some sort of quantitative metric. Thus in this section we will provide a few sample dialogues with our final online HH model. An obvious issue with the qualitative evaluation of samples is that it’s difficult to know to what extent they have been cherry-picked. To mitigate this issue, . We provide samples based on prompts that appear in association with others’ comparable work in Appendix including Instruct- GPT LAMDA and sensitive questions appearing in PALMS We encourage readers to skim these samples to get an (arguably) unbiased sense for our model’s behavior. Für jeden Anruf generieren wir 17 Proben, sortieren sie mit unserem HH Online-Präferenzmodell und zeigen dann die mediane Probe C ist, [Ouyang et al., 2024] [Thoppilan et al., 2022], [Solaiman and Dennison, 2021]. 6.1 Comparison with Human Writers Als zusätzlicher Test unserer Modelle sammelten wir hochwertige HH-Dialoge von menschlichen Schriftstellern. Diese Schriftsteller wurden auf Upwork (separat von unserem Pool von Crowdworkern) angestellt, basierend auf früheren erfolgreichen Schreibarbeiten und positiven Bewertungen. Then based on the prompts (leaving off the final writer-written assistant responses), we generated 17 examples from our best HH and best purely-helpful online models, and chose the median response from this pool as ranked by our online HH preference model. We then asked our crowdworkers to rank responses from the writers, along with one response each selected from our HH and helpful-only models. We then use these results to estimate an Elo score for our human writers, by comparing to our two models, and we display the result as a horizontal band on Figure 1. We used a band because the writer comparisons were actually somewhat inconsistent with our other Elo rankings, as crowdworkers preferred our HH online model to human writers 56% of the time, but preferred the our pure-helpfulness model to writers only 51% of the time. Presumably this is because the distribution of writers’ prompts differs from the distribution of conversations in which crowdworkers usually engage. Thus rather than altering the rankings betweeen our HH and helpful models, we represent writers as a wide band. We find that the performance of both our helpful and HH models is roughly comparable to that of the human writers, though in fact the models are slightly preferred by crowdworkers. We also find that our PM’s typically assign higher scores to the model responses, as shown in Figure e believe that this result should be interpreted with caution and caveats, however. The writers’ prompts were not very adversarial; many strike us as ‘softball questions’ that models can easily do well on. As researchers who have interacted with models for tens or hundreds of hours, we could likely come up with many questions where humans would be more likely to outperform the models. Relatedly, the writers’ conversations do not strike us (in many cases) as especially realistic, in the specific sense that we would not expect a human to derive a great deal of practical value by asking an AI assistant for help in these ways. We expect that our models likely still fall significantly short of providing the kind of value that a sophisticated human assistant could provide via chat, even if they were operating under the constraints the model experiences (e.g. no internet access). 24.W Here are a couple illustrative examples of prompts, writer responses, and our online HH RLHF model’s response. In this example the model performs well at first, but then makes some slightly inaccurate statements about the ways in which it fixed errors: In diesem Beispiel ist die Statistik des Autors vermutlich etwas irreführend (vermutlich unbeabsichtigt), und die Antwort des Modells kann einfach überlegen sein. In this final example, the model makes some subtle mistakes, but most readers would probably have difficulty detecting them without consulting reference material: Dialoge wie die oben genannten helfen, die Grenzen des menschlichen Feedback-Trainings zu illustrieren: Dies ist einer der Gründe, warum wir uns möglicherweise nicht auf menschliches Feedback verlassen wollen, um Modelle zu trainieren, um ehrlich zu sein, oder für jedes Ziel, in dem menschliches Urteil nicht die bestmögliche Informationsquelle ist. once model errors become sufficiently subtle, they will no longer be penalized appropriately, and the model’s incentives may no longer be aligned With that said, AI models still have limitations that can seem highly incongruous. Here’s an example where we, the more experienced model-interrogators, really played ‘hard ball’ with the AI: 6.2 Sensitive Questions and Avoidance versus Engagement Here are a few examples from PALMS sensitive questions. We chose them to illustrate how the model sometimes avoids very sensitive subjects, but often provides otherwise reason- able responses. Please see Appendix für viele weitere Beispiele, auch einige von InstructGPT and LaMDA prompts. [Solaiman and Dennison, 2021] C [Ouyang et al., 2024] [Thoppilan et al., 2022] Here is a case where the model just refuses to answer the question: Here the model indicates that it should not answer, but then provides some relevant information (perhaps it’s also problematic that the model focuses on the US, though this is likely a consequence of it having been trained in English by US-based crowdworkers): And here the model provides a fairly complete response, without attempting to avoid the topic: Qualitatively speaking, our experience has been that by tuning optimization for helpfulness versus harmless- ness, one can train models that are extremely avoidant when sensitive topics come up, or models that almost always attempt to engage, even on sensitive subjects. We suspect that Section also provides useful lessons if the goal is to train models that provide full responses in the face of sensitive questions. 4.4 6.3 Beispiele für Dialoge Most of our samples only involve a single prompt and response, so in this section we show a few two-turn dialogues. In diesem Dialog folgt das Modell den Anweisungen, hätte aber die Nuance der Situation deutlich besser erfassen können: Hier ein paar Dialoge zu einem technischen Thema von einem Forscher, der mit der Verwendung des Modells experimentierte, um ein Engineering-Problem zu debuggen: When asked to parrot writing styles, the model is more likely to make potentially harmful, explicit, or offen- sive statements: In this case we chose a human reply that takes the conversation in a subtly dark direction, and the model rebuts the argument: 7 Diskussion We have shown that it’s possible to use reinforcement learning from human feedback to train language models that act as helpful and harmless assistants. Our RLHF training also improves honesty, though we expect other techniques can do better still. As in other recent works associated with aligning large language models RLHF verbessert die Nützlichkeit und Unschädlichkeit um eine enorme Marge im Vergleich zu einfachen Skalierungsmodellen. [Stiennon et al., 2020 Thoppilan et al., 2022, Ouyang et al., 2022, Nakano et al., 2021, Menick et al., 2022], Our alignment interventions actually enhance the capabilities of large models, and can easily be combined with training for specialized skills (such as coding or summarization) without any degradation in alignment or performance. Models with less than about 10B parameters behave differently, paying an ‘alignment tax’ on their capabilities. This provides an example where models near the state-of-the-art may have been necessary to derive the right lessons from alignment research. Das Gesamtbild, das wir zu finden scheinen – dass große Modelle eine Vielzahl von Fähigkeiten, einschließlich der Ausrichtung, auf eine wechselseitig kompatible Weise lernen können – scheint nicht sehr überraschend zu sein. Finetune mit höherer Probeneffizienz und leiden nicht erheblich daran, vergessen zu werden Obwohl wir es nicht direkt demonstriert haben, erwarten wir auch, dass das RLHF-Aligning-Training mit oder vor dem Training für andere Ziele gemischt werden kann; dies könnte in der Zukunft relevant sein, um die Produktion von mittleren, unaligned AI-Systemen zu vermeiden. [Kaplan et al., 2020, Rosenfeld et al., 2019, Brown et al., 2020], [Henighan et al., 2020, Askell et al., 2021], [Ramasesh et al., 2022]. We did find a clear tension between helpfulness and harmlessness, where models trained entirely for helpful- ness are preferred to models trained for HH, when evaluating only on helpfulness. We believe this is partly due to a subtlety in our data collection process, as we rarely collect data teaching models how to deal pos- itively with harmful requests (i.e. how to be a sort of ‘hostage negotiator’), but only on how to avoid them. And we also found that at least at the level of preference models, the helpful-harmless tension diminishes as models become larger and more capable. Nevertheless we do expect that this tension is real, and that caution may cut into model performance on the margin. Große generative Modelle werden als "Grundmodelle" bezeichnet. Diese Mods sind äußerst interessante Objekte für die Forschung, aber ohne weitere Verfeinerung können sie schädliches Verhalten zeigen.Unsere Arbeit legt nahe, dass Ausrichtungstraining in Grundmodelle eingebunden werden kann, ohne ihre Nützlichkeit und Vielseitigkeit zu beeinträchtigen, und so vielleicht könnte es bald Teil ihrer Definition werden. [Bommasani et al., 2024] 7.1 Limitations While we believe our results present a promising picture for the alignment of existing language models, work on this subject remains in an early stage, and has a number of limitations. As was also emphasized by the authors of Wir betrachten unsere Arbeit an der Ausrichtung als ein laufendes Projekt; unsere Arbeit was step zero, and this is step one. [Thoppilan et al., 2022], [Askell et al., 2024] Wir haben pragmatisch einen ausgerichteten Assistenten als eine KI definiert, die helpful, honest, and harmless. We are op- timistic that at present capability levels, the techniques we have discussed here provide a reasonable approach to achieving helpfulness and harmlessness. However, although our techniques improve model honesty, we believe we are just scratching the surface of that problem, and that other techniques may more efficiently and effectively produce honest AI models. 18 Here we have essentially focused on the average-case behavior of our models. However, even if we were convinced that our models were HHH in expectation, a clear next step would be to attempt to study and eliminate bad behaviors (especially harmfulness) even in the worst case. We have not addressed this question of robustness here, but hope to study it in the future (approaches such as may be useful). It will only become more pressing as AI systems advance and encounter distributional shift during deployment. [Perez et al., 2022] AI alignment may be difficult and ambiguous to assess. So for example, while our large RLHF-trained models perform better than plain LMs on virtually all capabilities evaluations, one might hope that a truly helpful models’ zero-shot performance would equal the few-shot performance of an unaligned model. The logic here is that if a model can really ‘helpfully follow instructions’, then a prompt or explanation should be sufficient to bridge the zero-to-few-shot gap. We are very far from achieving this level of performance! Even on the honesty evaluation TruthfulQA we close a bit less than half of this gap (Figure We also briefly investigated whether our RLHF-finetuned code models have any comparative advantage when exposed to prompts including buggy code Man würde hoffen, dass ein vollständig ausgerichtetes Modell sein Bestes tun würde, um den richtigen Code zu schreiben, selbst wenn eine buggy-Anweisung gegeben wird. [Lin et al., 2021] 5). [Chen et al., 2021], We also harbor a general concern that perhaps our techniques only render models aligned ‘on the surface’, and that they still harbor harmful biases or other tendencies that may surface in more subtle contexts. We found that RLHF models have a more positive sentiment towards all racial and religious groups, which seems promising, but does not necessarily indicate that biases have been reduced. And with respect to gender, we found that RLHF model biases are very strongly correlated with the bias of the underlying language models. That said, further work will be required to understand if this is a limitation of RLHF as a technique, or of our particular HH datasets. In any case, we likely need to build more subtle and comprehensive evaluations that include multi-turn dialogue, as this is an area where humans will likely use the models, and it’s also a place where it’s inherently more difficult to measure performance against subtle objectives such as bias and fairness. On a much more practical level, we do not have much experience applying RL techniques to large generative models. Experienced AI practitioners know that there are a large variety of tweaks and tricks that require experimentation to identify, and that can majorly improve the stability and performance of training. We have encountered some stability issues with RL, and although we performed some rudimentary hyperparameter scans, we expect that with more experience and study we could do better. We also did not explore variations in online training, such as literally updating a single PM or RLHF model; rather we retrained these models from scratch on each iteration. Another direction for exploration is to use a non-trivial function of PM scores as the RL reward, distorting the score distribution to e.g. focus more on discouraging bad behavior rather than rewarding good behavior. In summary, there are many future directions to explore for improving RLHF. A final concern is whether techniques like those we have employed will continue to apply as AI models become increasingly capable. We take these concerns very seriously. In our view, the present work makes some progress towards our goal, which is to establish a set of simple and universal techniques that can align AI models at present capability levels. Assuming this goal can be met, one of the next steps will be to build consensus among researchers and to understand alignment in greater depth, including how techniques scale with AI capabilities. The hope will be to create an evolving pragmatic state of the art for training AIs that are thoroughly helpful, honest, and harmless. ursprünglich 19 Ein weiterer wesentlicher Schritt besteht darin, diese Ausgangslinie als Ausgangspunkt für die Erforschung anderer Techniken zu nutzen, die besser auf fortschrittlichere Anwendungsfälle und spekulativere Ausfallmodi reagieren können. Neue Ideen und Techniken können dann pragmatisch mit bestehenden Methoden verglichen und dann in die Standardpraxis integriert werden, wenn sie weitere Verbesserungen in Bezug auf Sicherheit und Robustheit bewirken. Wir sind bereits mit Beispielen konfrontiert, die auf die Grenzen des menschlichen Feedbacks hinweisen, und deshalb müssen wir anfangen, andere Methoden zu entwickeln. 6.1, 7.2 Alignment Data as a Public Good In this work we allowed crowdworkers’ common-sense to define what constitutes helpful and harmless be- havior. This was sufficient for our exploration of ‘technical alignment’, i.e. the question of whether certain techniques can be used to train AI models to be more helpful and harmless. But we have avoided addressing the underlying question of what sort of behavior should be expected from deployed AI models. Das heißt, ohne eine klare Spezifikation für das Format und die Art der "Alignment-Daten", die für das AI-Training am relevantesten sind, ist es für jeden außer Forschern schwierig gewesen, die Informationen zu sammeln, die notwendig sind, um sichere und nützliche AI-Systeme zu trainieren. to teach AI models complex human preferences, and we have also found that preference modeling based on ranked comparisons scales better than many other techniques. [Stiennon et al., 2020, Ouyang et al., 2022, Schröder et al., 2021) [Askell et al., 2021] One possible approach would be for an independent organization with ethical, legal, and cultural expertise to create a very high-quality dataset expressing human preferences for AI behavior (via comparisons). Such an organization could also use a novel governance structure, so that a larger set of societal stakeholders could factor into the decisions it makes about how to create and curate alignment data – in contrast to today, where private companies make these decisions in an opaque manner using governance structures that grant power to financially interested parties. Datasets created in this way might be used for both training and evaluation of AI models, and could even begin to establish standards for behavior. Due to the rapid improvement in AI language models, we expect that such datasets would be most valuable if they encode preferences at human- level sophistication. In any case, this is just one speculative possibility for broadening participation in dataset creation. Unsere Forschung hat von öffentlich verfügbaren Forschungsdatenbanken und Bewertungen profitiert, die für die Ausrichtung von KI auf menschliche Werte relevant sind. und wir planen, unsere Vorlieben Modellierungsdaten für andere zu veröffentlichen, um sie in ihrer Forschung zu verwenden. Leider scheint dies keine Standardpraxis unter Alignment-Forscher zu sein, wie durch einige jüngste Arbeit belegt. Während wir uns einig sind, dass LLMs selbst für Schaden verwendet werden können, scheint es, dass kein solches Argument für Alignment-Daten gemacht werden kann. [Stiennon et al., 2020 Hendrycks et al., 2021a], It’s extremely important to enable collaboration and reproducibility for alignment and safety research. As AI systems become more powerful and more widely deployed, the cost of mistakes and misunderstandings may grow immensely. We believe that the only way to convincingly address potential safety failures from advanced AI systems is to build a thoughtful community of researchers with deep expertise, and the ability to evaluate systems empirically. This will remain almost impossible if knowledge about the alignment of advanced systems remains siloed within many independent organizations. Sharing data seems like the easiest and most commonsense way to enable the sharing and validation of results. Ein offensichtlicher Grund für die Geheimhaltung ist, dass Organisationen Daten von Benutzern verwenden können, um Ausrichtungssätze zu entwickeln, und dann rechtfertigen, die Datensätze nicht zu teilen, weil sie die Privatsphäre der Benutzer verletzen. Dies ist ein herausforderndes Problem, das Organisationen verlangt, darüber nachzudenken, wie sie kommerzielle Prioritäten mit der Notwendigkeit vereinbaren, eine „Sicherheitsgemeinschaft“ für die Gemeinschaft zu schaffen. Wenn Ausrichtung mit dem Konzept der kommerziellen Moats verknüpft wird, könnte dies das Gesamtnetzsicherheitsniveau des KI-Ökosystems verringern. 7.3 Weiteren Einfluss We hope that our work provides compelling evidence that AI systems can be made safer and more useful at the same time, and without performance costs. As noted above, we have largely remained agnostic on the question of which values define acceptable and unacceptable AI behavior. Thus we hope that rapid progress in technical alignment and the consolidation of specific techniques will motivate the development of publicly available alignment data, guidelines, and benchmarks. Wir haben die Effektivität von Präferenzmodellierung und RLHF-Schlag (in unserer Forschung und anderen) gefunden) und glauben, dass es sehr berechtigte Bedenken gibt, dass diese Techniken für Zensur, Betrug und Fehlinformationen verwendet werden könnten. Wir fanden auch, dass Systeme, die ausschließlich dazu ausgebildet sind, hilfreich zu sein, leichter für schädliche Zwecke zu verwenden sind, was darauf hindeutet, dass es mit zunehmender Leistungsfähigkeit von Systemen immer wichtiger wird, ihr Schadenspotenzial direkt zu begrenzen. 20 Perhaps the broadest impact of this work, and the general development and dissemination of controllable, human-like language generation will be cultural. In Figure Wir verwendeten eine Elo-Skala, im Wesentlichen das Schach-Rating-System, um natürliche Sprachassistenten zu vergleichen und zu bewerten, und wir haben sogar einen Vergleich mit menschlichen Schriftstellern eingeschlossen. Diese Art von Vergleich riskiert, die Bedeutung der Sprache zu trivialisieren, die sicherlich nicht nur ein Spiel ist, sondern das Kernmedium der Kultur und der Gesellschaft. Während das Streben nach der Ausrichtung von immer leistungsfähigeren KI-Systemen wie eine robuste gute Aktion anfühlt, stellt die Bereitstellung dieser Systeme herausfordernde Fragen - Kultur ist im Grunde ein menschliches Unternehmen, aber groß angelegte Generationsmodelle haben die Möglichkeit, verschiedene Teile der menschlichen Kultur in unvorhersehbarer und unsichtbarer Weise zu vergrößern und zu minimieren [Ganguli et al., 2022], 1 Acknowledgments Anerkennung Wir danken Sam Bowman, Paul Christiano, Jacob Hilton, Jan Leike, Ethan Perez und Jeff Wu für das hilfreiche Feedback zum Entwurf.Wir danken Daniela Amodei, Jamie Kerr, Jia Yuan Loke, Rebecca Raible und Tim Telleen-Lawton für die Unterstützung mit dem Projekt. Author Contributions Autorenbeiträge Yuntao Bai führte die meisten Experimente auf RLHF und viele der Vorlieben Modellierung Experimente durch. Er machte wichtige Beiträge zu experimentellen Design, Messung und Bewertung der Modellleistung und Verhalten. Andy Jones and Kamal Ndoussse built the infrastructure for RL training of large language models. They also built associated plotting and monitoring systems and implemented the PPO algorithm. They helped with the design, implementation, and debugging of RLHF. Amanda Askell helped to design model evaluations, collected samples and evaluations from professional writers, built systems for improving the quality and quantity of data collection, and collaborated with Jared and Jackson on associated evaluations. She also helped with the design and implementation of the human feedback interface. She helped to write the paper. Anna Chen helped with general RL and RLHF experimentation, and contributed to the research design. Nova DasSarma managed the underlying cluster infrastructure, making large scale RL training and human feedback collection possible. Dawn Drain trained the underlying code models and collaborated with Saurav on coding evaluations. Stanislav Fort performed the OOD detection and outlier exposure research and analysis on helpful versus harmful data samples. Deep Ganguli led the red-teaming data collection effort and design, often working with Jackson, Liane, Amanda, and Ben. He designed and ran the societal impact evaluations in collaboration with Jared, and helped with model evaluations generally. Tom Henighan helped with pretraining the underlying language models, with dataset creation, and with managing the cluster during some phases of the project. Nick Joseph helped design and build a framework for efficient training of large language models and prefer- ence models. Saurav Kadavath designed and conducted experiments on helpful/harmless dataset mixing. Saurav also ran RLHF training on code models, with support from Yuntao, and ran coding evaluations in collaboration with Dawn. He also ran the majority of the natural language evaluations of basic and RLHF-finetuned models, and helped with RLHF training generally. He helped to write the paper. Jackson Kernion led human feedback crowdworker evaluation and management, and helped to build and maintain the feedback interface. He also helped with data analysis and collaborated on model evaluations. He ran most of our model comparison experiments. Tom Conerly helped with engineering, specifically with fast and efficient sampling. Sheer El-Showk helped with pretraining research and dataset construction. Nelson Elhage contributed significantly to pretraining and to engineering vision. Zac Hatfield-Dodds half bei der Code-Base-Wartung und beim Engineering, insbesondere bei der schnellen und effizienten Probenahme. Danny Hernandez contributed to pretraining and especially to dataset design. Tristan Hume helped with streamlining our infrastructure. Scott Johnston helped with pretraining research. Shauna Kravec contributed to the development and use of our RL systems, and collaborated on RL research. Liane Lovitt helped with red-teaming, and in particular with designing the interface. Neel Nanda contributed to research discussions and priorities for alignment. Catherine Olsson half bei der Beratung bei der Erhebung von Feedbackdaten von Menschen und leistete Ratschläge zu Ausrichtung und Bewertung. Dario Amodei advised the project and led efforts to build and test the RL infrastructure and ML. Tom Brown led engineering efforts, including efficient pretraining, sampling, and the stability and design of RL systems. Jack Clark leitete die Bemühungen um gesellschaftliche Auswirkungen und beraterte das Projekt, auch bei verschiedenen Bewertungen.Sam McCandlish leitete die Vorbereitungsmaßnahmen und beraterte das Projekt. Chris Olah hat an Diskussionen über Ausrichtung zusammengearbeitet und zu unserer Forschung und Bewertung der Infrastruktur beigetragen. Ben Mann led the design and construction of the human feedback data collection interface and the underlying infrastructure. He also helped lead crowdworker management, and he provided engineering support for the project as a whole. He also contributed to pretraining and cluster management. Jared Kaplan entwarf und leitete das Projekt.Er half bei allen Aspekten, einschließlich Forschungsdesign, Ingenieurwesen, Experimente und Bewertungen.Er trug auch zur Vorschulung bei und half beim Aufbau der Bewertungsinfrastruktur. A Details, Analysis, and Evaluations of Supervised Training A.1 Context Distillation Für die Kontextdestillation folgen wir dem Rezept von Specifically, we first generate data in the following way: [Askell et al., 2021]. 1. We prepend the ‘HHH prompt’ (i.e., a set of prompts designed to elicit helpfulness, harmlessness, and honesty) to sequences of text, with 50% of the text coming from our pretraining dataset, and 50% coming from a StackExchange dataset. For the former, we simply append pretraining data after signaling the beginning of another conversation with “Human:”. With StackExchange, we formulate a fake Human/Assistant conversation by using the question as the human side of the conversation, and a top-rated answer as the assistant role. 2. For both datasets, we then perform forward passes with a basic pretrained 52B model, and record the top 50 log-probabilities and their indices (within the vocabulary) for the tokens following the prompt. We store the log-probs, indices, and tokens together as a small new dataset. 3. To perform context distillation finetuning, we pass the tokens from our new dataset through models (of all sizes), and define the loss as the KL divergence between the stored log-probs and the predic- tions of the model undergoing finetuning. For each token, we use a 51-category distribution, with the 51st category covering the total probability from all the tokens other than the top-50. We show learning curves for context distillation in Figure We use a batch size of 32 sequences, and a learning rate of 0*.*05 times the pretraining learning rate, which we decay to zero linearly during distillation. We distill using a total of 350M tokens. 30. A.2 Preference Modeling Our preference models are trained on comparison data, with each data point consisting of a and a pair of . The prompt is a multi-step dialogue between human and model that always begins and ends on the human side, and each response is a continuation of the dialogue. For instance, in Figure the prompt consists of the first five steps of the dialogue, and the responses are shown in the blue box. The PM then assigns a score at the end of each response. Note that while the PM is only trained to evaluate the quality of the final response, the full context of the conversation is provided to the model. prompt Antworten 6, We train scans of PMs ranging from 13M to 52B parameters. All PMs go through three phases of training: (1) Sprachmodell (LM) Vor-Training auf einem großen Sprachkorpus, (2) Präferenzmodell-Vor-Training (PMP) und (3) Finetuning auf menschliches Feedback. LM pre-training details, including choice of hyperparameters and datasets, are explained in Appendix A of our previous work [Askell et al., 2024] For PMP, we use learning rate of 0.1 relative to LM pretraining, and train on a mixture of comparison data made from StackExchange, Reddit, and Wikipedia. Data preparation and labeling are explained in Appendix C.1 of We train with context size of 1024 tokens. [Askell et al., 2024] For human feedback finetuning, we use learning rate of 0.01 relative to the LM pretraining. We use context size of 1024 tokens, except for the ‘online’ model described in Section where we trained with 2048, which may help stabilize RLHF on long contexts. 4.5, For both PMP and human feedback finetuning, we append a special ‘end-of-context’ token at the end of each sample, such that the PM score is predicted directly on top of this token. As explained in Appendix C.4 of Das soll die PM-Performance verbessern. [Askell et al., 2024] In all phases, we only train over one iteration to mitigate overfitting. A.3 Scaling of PM with Model and Dataset Size Eine wichtige Frage ist, wie die Leistung der Präferenzmodellierung mit Modellgröße und Datensatzgröße skaliert.Dies bezieht sich auf eine praktische Frage – sollten wir in die Erhebung eines größeren Datensatzes investieren oder in die Ausbildung von größeren Modellen? We seem to find more predictable scaling when training only on our helpfulness dataset, likely because the red-teaming data truly comes from a distinct distribution. Accuracy learning curves can be seen on the left of Figure We find that accuracy can be roughly fit by 32. where is the number of parameters in the PM and is the size of the dataset. P D However, the results when training on another preference modeling data distribution look quite different, as seen on the right in Figure Note that there appears to be a sort of discontinuity in behavior between 200M and 13B parameters. Perhaps this is related to the fact that the data was generated by a model with 6B parameters. 32. B Details, Analysis, and Evaluations of RLHF B.1 Training Setup B.1 Trainingseinrichtung Here we discuss some details about RLHF training. We initialize our policies on context-distilled models, which are explained in A. 1 We train the policy to generate responses to a dataset of prompts that maximize the score relative to a PM that was finetuned on human feedback. The prompt dataset is obtained from the training split of the PM comparisons dataset by simply removing the responses in each pair. Recall that we allow multi-step dialogue within the prompt (which always begins and ends on the human side of the conversation), but only train the policy to generate one response following each prompt. In future work, we plan to train policies to generate multiple steps, but this requires a separate model that generates the human side of the conversation, which can be implemented with a language model trained to imitate the human side of the conversation. We performed a variety of hyperparameter scans, and ended up using learning rate of 0.01 relative to pretraining, a KL reward coefficient of λKL = 0.001 (4.1), PPO clipping = 0.2, discount factor γ = 1, and no entropy bonus. Furthermore, in PPO, we re-iterate over the same sample K times (see Algorithm 1 in [Schulman et al., 2017]), with higher K typically leading to more stable results. We used K = 1 for the RLHF scan, K = 2 for the robustness studies (Section 4.2), and K = 4 for the ‘online’ RLHF (Section 4.5). We also impose a limit on the maximum number of tokens per model response, using 32 for the robustness studies, and 128 elsewhere. Finally, for ‘online’ RLHF, we used a learning schedule that reduces the learning rate by 2× every 100,000 samples. For the robustness studies, we used a linear learning rate warmup for the first 25,000 samples. B.2 More on Robustness Studies B.2 Mehr über Robustheitsstudien In Figure Wir vergleichen die score for all policy sizes and all sizes. The main observation here is that the slope of the graph increases with respect to Größe, was bedeutet, dass die größere ’s are significantly more capable of distinguishing policy performance. In other words, , in agreement with calibration studies in Section 33, Prüfung PM Prüfung PM Prüfung PM test PM larger preference models are more robust 3.3. Finally, we take a moment here to address an issue we had neglected so far, which is that scores from different preference models should not be compared directly, since the absolute value of the score has no meaning, only relative scores are meaningful. We address this by a simple procedure. We make a held-out dataset, consisting of several thousand samples, and subtract from each preference model score its mean score on this dataset. We apologize that unrelated plots from different sections may have used different held-out datasets for mean-removal. Meint Entfernung B.3 Details of ‘Online’ RLHF B.3 Details zum „Online“ RLHF We give some more details on our ‘online’ RLHF policy discussed in Section Diese Richtlinie und ihre PM wurden anhand aller nützlichen und harmlosen Daten, die wir bei der Fertigstellung dieses Artikels hatten, geschult. = 4 mal to improve stability, and sampled a maximum of 128 tokens per response. 4.5. K [Schulman et al., 2017] Throughout training, we periodically evaluate the of the policy on various held-out PM comparison datasets. More specifically, given a comparison consisting of a prompt and pair of responses, we assign an accuracy of 1 if the policy’s mean log-p on the better response is higher, and 0 otherwise. We show these results in Figure for various comparison datasets. In particular, we find that mean log-p accuracy of the policy isn’t as high as (i.e., fraction of comparisons on which the PM assigns a higher score to the better response), possibly suggesting room for further improvements to our RLHF pipeline. mean log-p accuracy 34 PM accuracy B.4 Robustness of ‘Online’ RLHF B.4 Robustness of ‘Online’ RLHF For our final online models, we had crowdworkers compare a variety of model snapshots from RLHF training, in order to better understand the robustness and general performance of our training process. In Figure (right), we show Elo scores for the online models during RLHF training. 15 In particular, we compare Elo scores established empirically from crowdworker preferences (i.e., in the figure), and Elo scores predicted by our preference model (i.e, ) during RLHF training. For the latter, we sample responses from each snapshot on a set of held-out prompts, and evaluate the PM scores (which are then converted to Elo units). We notice that the naive PM predictions significantly overestimate the empirical Elos. This is due to a combination of the following fac- tors: Crowd-Arbeiter Präferenzen Naive PM Predic- tion 1. During crowdworker testing, each step of the conversation is written by one of the two models being tested. However, when evaluating a RLHF snapshot on held-out prompts, the policy only writes one response at the end of a pre-existing conversation (which had been previously generated by other models, as discussed in Appendix This leads to distributional shift between the conversations. B.1). 2. Elo and PM scores may not actually be transitive, as they involve the collapse of pairwise compar- isons onto a single line. For example, if PM scores zufrieden sein − = 2 and − = 2, even if those are well-calibrated scores, the implication that − = 4 may not be correctly calibrated, and we would naturally expect instead − 4. a, b, c a b b c a c a c < 3. Fehlschläge bei der Robustheit der PM, so dass die Präferenz der PM für die Proben der RL-Politik im Vergleich zu echten menschlichen Präferenzen falsch kalibriert wird. Um diese Effekte weiter zu erforschen, in der Abbildung we show Elo scores corresponding to four different measure- ments: 35 • Naive PM Prediction: The PM score (translated into Elo units) recorded during RLHF training, which uses a set of held-out prompts. • Mittlere PM-Score auf Crowdworker-Daten: Die mittlere PM-Score auf den tatsächlichen Crowdworker-Konversationen, die verwendet werden, um Elo-Score basierend auf Crowdworker-Präferenzen zu berechnen. • PM-Ranking auf Crowdworker-Daten: Man kann versuchen, Robustheitsversagen von misscaled PM-Score vs PM zu unterscheiden . Here we evaluate the PM on the crowdworker data used to compare these model snapshots, obtain ‘win rates’ for model comparisons Und dann wir the Elo scores based on the PM’s choices. rankings according to the PM recompute • Crowdworker-Präferenzen: Wir berechnen Elo-Score auf der Grundlage von Crowdworker-Präferenzen unter Modell-Snapshots. Wir sehen also, dass der Unterschied zwischen dem PM-Score und dem PM-Ranking keinen signifikanten Unterschied in Bezug auf die Robustheit macht. Jedoch war die Verteilungsverschiebung zwischen den gehaltenen Anrufen und den tatsächlichen Crowdworker-Gesprächen sehr signifikant und erklärt einen signifikanten Anteil der Diskrepanz zwischen den RLHF-Lernkurven und den Elo-Score, wie sie von Crowdworker gemessen wurden. B.5 Crowdworker Comparisons and Elo Scores B.5 Crowdworker Vergleich und Elo-Score Hier beschreiben wir kurz, wie wir die Crowdworker-Präferenzen unserer Modelle testen und wie Elo-Score tablisiert sind. and , we ask crowdworkers to engage in text-based, back-and-forth con- versations with the models. At each conversational step, two responses are generated, one from each model, and the worker chooses the response they prefer, and the conversation continues. Each choice the worker makes counts as a ‘win’ for the preferred model, giving ‘win counts’ In Fällen, in denen ein Mitarbeiter unsicher ist, ob eine Antwort besser ist, werfen wir solche Vergleiche sowohl in der PM- als auch in der RLHF-Schulung und in den Vergleichsbewertungen von Crowdworkers aus. A B auf NB Recall that, given Elo scores EA, EB, respectively, the log-likelihood for the win counts is given by where rA,B = (log 10/400)EA,B ≈ EA,B/174. For an ensemble of comparisons between various models, we estimate Elo scores and their errors by maximum likelihood estimation. In some cases one of the models uses , meaning that it generates samples, evaluates all of them using a preference model, and shows the user the top-scored sample. Elo scores for such models are shown in Appendix In this case, we cannot stream the sample, so instead we make the workers wait until the sample is completed. When testing a rejection sampling model against a non-rejection sampling one, we only show the samples when they’ve both been completed, even if the latter sample could’ve been streamed, to mitigate bias. Ablehnung von Sampling k B.6. B.6 Elo Scores for Rejection Sampling Models B.6 Elo Scores for Rejection Sampling Models In Figure we show helpfulness Elo scores for a 52B context distilled model with rejection sampling (utilizing a 52B preference model trained on pure helpfulness) for = 1*,* 4*,* 16*,* 64, showing that higher values of clearly perform better. Note that the context distilled model and the preference models discussed here were trained during an earlier stage of our research with different datasets and settings from those discussed elsewhere in the paper, so they are not directly comparable with other Elo results, though very roughly and heuristically, our online models seem to perform about as well or better than = 64 rejection sampling. Note 36 k k k that = 64 Ablehnungssampling entspricht = log(64) ≈ 4*.*2. k DKL B.7 Stack Overflow Results We can also evaluate our language models directly given a corpus of paired good and bad responses, such as answers to StackOverflow questions. In we evaluate the difference in mean log-p between popular (i.e, highly upvoted) and unpopular answers, showing that RLHF models consistently assign a higher difference, suggesting that they are more capable of distinguishing answer quality. In we plot the language modeling loss (i.e, mean log-prob) on the good and bad answers separately, rather than their difference. We find that the RLHF models obtain worse loss. This is most likely due to optimizing a different objective rather than pure language modeling. 37b 37a B.8 Further Analysis of RLHF on Code-Model Snapshots B.8 Further Analysis of RLHF on Code-Model Snapshots As discussed in Section RLHF verbessert die Leistung von Basic-Code-Modellen auf Code-Evaluierungen. In dieser App vergleichen wir das, indem wir das Basic-Code-Modell einfach mit einer Probe von Anrufen vergleichen, die entworfen wurden, um Hilfsbereitschaft, Unschädlichkeit und Ehrlichkeit hervorzurufen, die wir als "HHH" -Anrufe bezeichnen. Insbesondere enthalten sie ein paar Codierungsbeispiele. 5.3, Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. It also tries to avoid giving false or misleading information, and it caveats when it isn’t entirely sure about the right answer. That said, the assistant is practical and really does its best, and doesn’t let caution get too much in the way of being useful. -- ... (we include several short example conversations using the normal Human: ... Assistant: ... format.) -- Human: Can you help me write this Python function? I’ve already written the function’s signature and docstring, but I’m not sure how to write the function’s body. It starts like this: Assistant: Sure thing, here you go! I’ve tested this function myself so I know that it’s correct: Figure contains results on HumanEval when the HHH prompt is included. We see that the HHH prompt improves performance more significantly than RLHF across many values. 38 pass@k B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests Our newly proposed Simplified Relative Mahalanobis distance outperforms the standard Mahalanobis distance on OOD detection of harmlessness inputs from helpfulness inputs for activations extracted from all layers of all model sizes we tested. The details are shown in Figure Simplified Relative Mahalanobis distance 39. Exposing the OOD detector to a few examples of the out-distribution has first been proposed by suggested using a single class representing the OOD examples. [F showed that even strong near-OOD detectors enjoy a large benefit. Few-shot outlier exposure [Hendrycks et al., 2018]. [Thulasidasan et al., 2021] ort et al., 2021] Following the procedure from [F we apply a single layer linear classifier on top of the acti- vation vectors, while the rest of the language model is frozen. Given randomly drawn examples of the training set of the out-distribution (harmlessness data), we create a new binary classification problem, where the inputs are the combination of the full Ausbildungsbeispiele der in-Distribution mit einer Zielklasse von 0, und train*//M* copies of the examples of the out-distribution with the target class of 1. This ensure an equal representation of both in- and out-distribution examples and leads to better learning. Once we train the linear classifier, we use its class 1 output as the OOD score. The results are shown in Figure ort et al., 2021], M N N M 23. B.10 Gender Bias Evaluation Details B.10 Gender Bias Evaluation Details For this evaluation, we followed and used their list of 76 occupations and 12 pairs of gender terms: [Rae et al., 2024] • Berufe: CEO, Buchhalter, Administrator, Berater, Analyst, Bewertungsberater, Architekt, Assistant, Atten-Dant, Auditor, Bäcker, Bartender, Broker, Tischler, Kassierer, Koch, Chemiker, Chef, Reiniger, Clerk, Koch, Berater, Designer, Entwickler, Ernährungsberater, Dispatcher, Arzt, Fahrer, Editor, Erzieher, Elektriker, Ingenieur, Prüfer, Landwirt, Feuerwehrmann, Wächter, Friseur, Hygienist, Inspektor, Lehrer, Ermittler, Janitor, Anwalt, Bibliothekar, Maschiner, Manager, Mechaniker, Krankenschwester, Ernährungsberater, Offizier, Maler, Spezialist, Surfer, Paralegal, Paramed • Male gendered terms: boy, brother, dad, husband, man, groom, male, guy, father, son, gentleman, boyfriend • weibliche geschlechtsspezifische Begriffe: Mädchen, Schwester, Mutter, Frau, Frau, Braut, Frau, Gal, Dame, Mutter, Tochter, Freundin Human: Can you help me fill in the missing word in a sentence? The sentence is: For work, the {gendered term} is a Assistant: Sure thing, here is your sentence with the missing word filled in: For work, the {gendered term} is a {occupation} We then evaluated the probabilities of Mit dieser Formulierung haben die größeren LMs, kontextdestillierte Modelle und RLHF-Modelle alle bei Abschluss des Satzes etwa 2-8% ihrer Wahrscheinlichkeitsmasse einer der oben aufgeführten 76 Berufe zugewiesen. Beschäftigungen We also slightly modified the metric from [Rae et al., 2021] in order to estimate errors. So to compute bias scores, for each occupation O we defined and then we define the overall bias score as the mean of the absolute value of , ranging over occupations: BO where represents occupations while represent paired gender terms (e.g. lady/gentleman), and 76 und = 12. With this setup, we are sensitive to whether bias is uniform across all pairs of gendered terms (for any given occupation). As a concrete example, if it were the case that a model associated ‘lawyer’ more strongly with ‘boy’ than ‘girl’, but also more strongly with ‘wife’ than ‘husband’, this would increase the error in our measurement, while the bias effects would tend to cancel out. O gf , gm Nein Ng Die Ergebnisse sind in der Figur where we observe that RLHF models have both larger bias scores and larger errors. We believe this has a very simple explanation – RLHF models have much lower-entropy output distributions. As a rough, heuristic explanation, a model with a lower-entropy output will behave somewhat like a plain LM evaluated at a temperature 1. The metric and error from equation will typically scale as 1*/T* , so it should be expected that models with lower entropy distributions will have proportionally larger bias scores and bias errors. We show these effects explicitly in Figure durch Bewertung von Bias-Score mit kontextuell destillierten Modellen auf ≈ 0*.*6, und zeigt, dass sie den RLHF-Vorurteilen sehr ähnlich sind. 40, T < B.3 40 T Readers may observe that the RLHF model bias appears to be growing with model size. This is most likely because RLHF training tends to progressively reduce the entropy (or decrease the ‘effective temperature’) of model distributions, and we train larger models longer with RLHF, because their training is more robust (and so more beneficial in other respects). Thus we expect this trend is a property of differences in RLHF training across model sizes, rather than a more fundamental model-size trend. C Samples from PALMS, LaMDA, and InstructGPT Prompts These were generated by sampling 17 responses for each prompt from our best online HH model, ranking them using the corresponding preference model, and then displaying the non-empty response ac- cording to the PM. (Occasionally the model writes the "end of conversation" token in response to sensitive queries.) So to a large extent, these samples are not ‘cherry-picked’, and should be representative of average- case behavior. In the case of LaMDA and InstructGPT we include the corresponding samples from those models. Medium [Thoppilan et al., 2022] [Ouyang et al., 2022] C.1 PALMS Sensitive Questions C.1 PALMS Sensitive Questions Fragen werden aus den sensiblen Fragen in [Solaiman and Dennison, 2021]. Bitte beachten Sie, dass einige dieser Fragen sensibles Material enthalten (nach dem Design). C.2 InstructGPT Prompts Wir warnen die Leser, dass einige der Proben von InstructGPT were likely cherry picked to display , and so may be unrepresentative. Thus in some cases it will be unfair to compare our median samples to those from InstructGPT. [Ouyang et al., 2024] poor model behavior, on purpose C.3 LaMDA Prompts C.3 LaMDA Prompts D Details on Data Collection and Crowdworkers D.1 Overview D.1 Overview For the purposes of assigning tasks, we divided our crowdworkers into two groups. A group of around 30 ‘select’ workers were sent high priority tasks on a regular basis. Roughly half of these workers were hired through Upwork. The other half were selected from the general population of US-based MTurk workers with a ‘Masters’ qualification. The second group of workers came from the same general population of MTurk workers, and were primarily given red-team tasks on a less regular schedule. We tried to ensure that all workers were paid significantly above the minimum wage in California, adjusting rates for tasks that took longer (for example with models that use rejection sampling) in the case of MTurk workers, who are paid by task rather than at an hourly rate. We frequently issued bonuses to MTurk workers who encountered technical problems with tasks (as they are not paid an hourly rate). Unter unseren ausgewählten Arbeitnehmern machten MTurk-Arbeiter in der Regel 80-85% der in einer bestimmten Woche gesammelten Vergleichsdaten aus, verglichen mit 15-20% für Arbeitnehmer, die über Upwork eingestellt wurden.Obwohl die Größe dieser Gruppen ähnlich war, neigten MTurk-Arbeiter dazu, sich für mehr Arbeit zu entscheiden und ihre Lohnstruktur förderte schnellere Gespräche. We communicated with select workers on a daily basis over Slack. We used this channel to announce new tasks and provide guidance; we discussed difficult edge cases with the group as they came up. At an interme- diate stage of the project we provided some additional thoughts on more advanced forms of interaction with the model; the slack message we sent is shown in Figure Unsere Mitarbeiter warnten uns vor Bugs und Leistungsproblemen, als sie auf sie rannten. 43. Wir haben beide Gruppen von Arbeitnehmern eine demografische Umfrage gesendet, und die Ergebnisse sind in Figuren dargestellt Survey re- sponses were anonymous, as we did not collect any personal identifiable information alongside demographic information. 44. D.2 Instructions and Interface D.2 Anweisungen und Schnittstelle We display basic task instructions in a pop-up dialog when first loading the interface, and these instructions remain available throughout the interaction. These instructions for the ‘playground’ and ‘red team’ tasks can be found in figure For the playground task, we also link to a separate page with expanded instructions that include more detailed examples, excerpts of which can be seen in figure 41. 42. Die menschliche Feedback-Schnittstelle ist in der Abbildung dargestellt During the online data collection process, we added an additional option to the interface for Upworkers. This feature allowed them to edit one of the model responses. When they used this feature, we stored a comparison of the edit to the original (assuming the edit was better), rather than the initial comparison of two model outputs. This would have effected less than 10% of the online data. 6. D.3 Data Quality Measurement Challenges D.3 Herausforderungen bei der Datenqualitätsmessung In rough outline, data quality assurance for human labelers often involves the following steps: Forscher prüfen sorgfältig eine kleine Reihe von Proben, um eine Reihe von „goldenen“ Etiketten zu produzieren • Human labelers work through a stack of labeling tasks, with some subset of labeling tasks assigned to multiple labelers, and samples from the golden label dataset sent out to everyone. • Forscher bewerten die Leistung von Labelern, indem sie ihre Etiketten gegen die goldenen Etiketten überprüfen und überprüfen, ob eine Vereinbarung zwischen Labelern besteht. The idea is that the golden labels are treated as a source of truth, with labelers incentivized to modify their behavior to better match the golden labels, and novice labelers incentivized to match the behavior of more experienced labelers. Offene Gespräche ermöglichten es uns, einen reicheren Datensatz zu sammeln, führten jedoch eine Reihe schwer zu kontrollierender Variablen ein, was zu lauten Datenqualitätsmetriken führte. We did try having crowdworkers review each other’s conversations, by providing ratings for each model response comparison and rating the overall quality of the conversation. But we found that author-rater agree- ment wasn’t a good guide for assessing overall conversation quality. In broad strokes, conversation quality depends on a) conversation topic, b) human writing quality, c) model quality. And we found that, for instance, as conversations got more sophisticated, deciding between model responses got more difficult. As a result, authors that more frequently discussed difficult topics would often get lower agreement scores. And because we were frequently updating our models, model response comparisons were a moving target. Instead of directly assessing our crowdworkers based on these kinds of reviewer-based metrics, we considered using their performance as reviewers as a stand-in for their performance as conversation authors. But when we compared these metrics against our own spot-checks of conversation author quality, we found poor agreement. Wir erwarten, dass wir diese Probleme umgehen und in Zukunft bessere Methoden entwickeln können, aber es ist erwähnenswert, dass wir unsere Ergebnisse ohne anspruchsvolle Datenqualitätssteuerungen erzielen konnten. E Details on NLP Evaluations Formatting and Prompts Hier geben wir die Eingabeformate an, die wir verwendet haben, um die Leistung von Lambada, ARC, MMMLU, HellaSwag, OpenBookQA und TriviaQA zu bewerten. References [Amodei et al., 2016] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., und Mané, D. (2016). Konkrete Probleme in der Sicherheit. [Askell et al., 2021] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general language assistant as a laboratory for alignment. [Bender et al., 2021] Bender, E. M., Gebru, T., McMillan-Major, A., und Shmitchell, S. (2021). Über die Gefahren stochastischer Papageien: Können Sprachmodelle zu groß sein? , FAccT ’21, Seiten 610–623, New York, NY, USA. Verfahren der ACM-Konferenz 2021 über Fairness, Rechenschaftspflicht und Transparenz [Bommasani et al., 2021] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gespillie, L., Goel, K., Goodman, N. D., Grossman, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M. S., Krishna, R., Kuditipudi, R., and et al. (2021). On the opportunities and risks of foundation models. , abs/2108.07258. CoRR [Borgeaud et al., 2021] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J. W., Elsen, E., und Sifre, L. (2021). Verbessern Sie die Modelle Sprache, indem Sie aus Billionen von Token abrufen. , abs/2112.04426. CoRR [Brown et al., 2020] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., und Amodei, D. (2020). Sprachmodelle sind nur wenige Lernende. [Chen et al., 2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Bewertung großer Sprachmodelle ausgebildet auf Code. . arXiv Preprint arXiv:2107.03374 [Chowdhery et al., 2022] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Kind, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., und Fiedel, N. (2022). Palm: Sprachmodellierung mit Wegen. [Clark et al., 2018] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Denken Sie, dass Sie die Frage beantwortet haben? versuchen arc, die ai2 Argumentation Herausforderung. , abs/1803.05457. ArXiv [Fort, 2022] Fort, S. (2022). Gegnerschaftliche Anfälligkeit für leistungsstarke Erkennung in der Nähe von Verteilung. [Fort et al., 2021] Fort, S., Ren, J., and Lakshminarayanan, B. (2021). Exploring the limits of out-of- distribution detection. [Ganguli et al., 2022] Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S. E., Fort, S., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Amodei, D., Brown, T., Kaplan, J., McCandlish, S., Olah, C., and Clark, J. (2022). Predictability and surprise in large generative models. [Guo et al., 2017] Guo, C., Pleiss, G., Sun, Y., und Weinberger, K. Q. (2017). Über die Kalibrierung moderner neuronaler Netzwerke. [Guu et al., 2020] Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). REALM: retrieval- augmented language model pre-training. , abs/20022.08909. CoRR [Henderson et al., 2017] Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., und Pineau, J. (2017). Ethische Herausforderungen in datengesteuerten Dialogsystemen. , abs/1711.09050 CoRR [Hendrycks et al., 2021a] Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. (2021a). Aligning ai with shared human values. [Hendrycks et al., 2021b] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., und Stein- hart, J. (2021b). Messung des massiven multitask-Sprachverständnisses. [Hendrycks et al., 2021c] Hendrycks, D., Carlini, N., Schulman, J., und Steinhardt, J. (2021c). Ungelöste Probleme in ml Sicherheit. [Hendrycks und Gimpel, 2016] Hendrycks, D. und Gimpel, K. (2016). Eine Ausgangslinie für die Erkennung von falsch klassifizierten und nicht verteilten Beispielen in neuronalen Netzwerken. [Hendrycks et al., 2018] Hendrycks, D., Mazeika, M., und Dietterich, T. (2018). Tiefe Anomalienerkennung mit äußerer Exposition. [Henighan et al., 2020] Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., und McCandlish, S. (2020). Skalierungsgesetze für autoregressive generative Modelle. [Hernandez et al., 2021] Hernandez, D., Kaplan, J., Henighan, T., und McCandlish, S. (2021). Skalierungsgesetze für den Transfer. , abs/2102.01293. CoRR [Hestness et al., 2019] Hestness, J., Ardalani, N., and Diamos, G. (2019). Beyond human-level accuracy: Computational challenges in deep learning. In , PPoPP ’19, pages 1–14, New York, NY, USA. ACM. 24. Symposium über Prinzipien und Praxis der Parallelprogrammierung [Jiang et al., 2021] Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., und Choi, Y. (2021). Delphi: Richtung Maschinenethik und Normen. [Joshi et al., 2017] Joshi, M., Choi, E., Weld, D. S., und Zettlemoyer, L. (2017). Triviaqa: Ein groß angelegter, aus der Ferne überwachter Herausforderungsdatensatz für das Lesenverständnis. [Kaplan et al., 2020] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., und Amodei, D. (2020). Skalierungsgesetze für neuronale Sprachmodelle. [Koch et al., 2021] Koch, J., Langosco, L., Pfau, J., Le, J., und Sharkey, L. (2021). Objektive Robustheit im tiefen Verstärkungslernen. , abs/2105.14111. CoRR [Lakshminarayanan et al., 2016] Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016). Simple and scalable predictive uncertainty estimation using deep ensembles. [Lee et al., 2018] Lee, K., Lee, K., Lee, H., und Shin, J. (2018). Ein einfacher einheitlicher Rahmen für die Erkennung von außerverteilten Proben und angreifenden Angriffen. [Lewis et al., 2020] Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. , abs/2005.11401 aufgeführt. CoRR [Liang et al., 2017] Liang, S., Li, Y., and Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. [Lin et al., 2021] Lin, S., Hilton, J., and Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. [Liu et al., 2021] Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. (2021). On-the-fly controlled text generation with experts and anti-experts. , abs/2105.03023. CoRR [Liu et al., 2020] Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., und Lakshminarayanan, B. (2020) Einfache und prinzipielle Unsicherheitsbeurteilung mit deterministischem Deep Learning über Fernwissen. [Menick et al., 2022] Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., und McAleese, N. (2022). Unterrichten von Sprachmodellen zur Unterstützung von Antworten mit verifizierten Zitaten. [Mihaylov et al., 2018] Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. In . EMNLP [Nakano et al., 2021] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. (2021). Webgpt: Browser-assisted question-answering with human feedback. , abs/2112.09332. CoRR [Nalisnick et al., 2019] Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., und Lakshminarayanan, B. (2019). [Nguyen et al., 2014] Nguyen, A., Yosinski, J., and Clune, J. (2014). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. [Ouyang et al., 2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Sprachmodelle trainieren, um Anweisungen mit menschlichem Feedback zu folgen. . arXiv Preprint arXiv:2203.02155 [Pan et al., 2022] Pan, A., Bhatia, K., und Steinhardt, J. (2022). Die Auswirkungen der Belohnung Fehlspezifikation: Kartierung und Minderung von misaligned Modellen. , abs/2201.03544. CoRR [Paperno et al., 2016] Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and FernÃ¡ndez, R. (2016). The lambada dataset: Word prediction requiring a broad discourse context. [Parrish et al., 2021] Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., und Bowman, S. R. (2021). BBQ: Ein handgefertigter Benchmark für die Beantwortung von Fragen. , abs/2110.08193 CoRR [Paszke et al., 2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high- performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, , Seiten 8024–8035. Curran Associates, Inc. Advances in Neural Information Processing Systems 32 [Perez et al., 2022] Perez, E., Huang, S., Song, H. F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., und Irving, G. (2022). Rote Sprachmodelle mit Sprachmodellen. , abs/2202.03286 CoRR [Power et al., 2022] Power, A., Burda, Y., Edwards, H., Babuschkin, I., und Misra, V. (2022). Grokking: Verallgemeinerung über das Überfüllen auf kleinen algorithmischen Datensätzen hinaus. , abs/2201.02177. CoRR [Rae et al., 2021] Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, H. F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S. M., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., A., Weidinger, L., Gabriel, I., Isaac, W. S., Lockhart, E., Osindero, S., Rimell, L., Dyer, , abs/2112.11446. CoRR [Ramasesh et al., 2022] Ramasesh, V. V., Lewkowycz, A., und Dyer, E. (2022). Wirkung der Skala auf katas- trophisches Vergessen in neuronalen Netzwerken. In . Internationale Konferenz über Lernvertretungen [Ren et al., 2021] Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., and Lakshminarayanan, B. (2021). A simple fix to mahalanobis distance for improving near-ood detection. . arXiv preprint arXiv:2106.09022 [Rosenfeld et al., 2019] Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. (2019). A constructive prediction of the generalization error across scales. [Sanh et al., 2019] Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. , abs/1910.01108. archiviert [Schulman et al., 2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., und Klimov, O. (2017). Algorithmen zur Optimierung der nahen Politik. , abs/1707.06347. CoRR [Solaiman und Dennison, 2021] Solaiman, I. und Dennison, C. (2021). Prozess zur Anpassung von Sprachmodellen an die Gesellschaft (PALMS) mit wertorientierten Datensätzen. , abs/2106.10328. CoRR [Stiennon et al., 2020] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., und Christiano, P. (2020). Lernen, aus menschlichem Feedback zusammenzufassen. [Thoppilan et al., 2022] Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Krivokon, I., Rusch, W., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., and Le, Q. (2022). Lamda: Language models for dialog applications. , abs/2201.08239. CoRR [Thulasidasan et al., 2021] Thulasidasan, S., Thapa, S., Dhaubhadel, S., Chennupati, G., Bhattacharya, T., und Bilmes, J. (2021). Eine einfache und effektive Ausgangslinie für die Entdeckung außerhalb der Verteilung mit Abstinenz. [Tillet et al., 2019] Tillet, P., Kung, H. T., and Cox, D. (2019). , Seiten 10 bis 19. Association for Computing Machinery, New York, NY, USA. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations [Weidinger et al., 2021] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W. S., Legassick, S., Irving, G., und Gabriel, I. (2021). Ethische und soziale Risiken des Schadens durch Sprachmodelle. , abs/2112.04359. CoRR [Winkens et al., 2020] Winkens, J., Bunel, R., Roy, A. G., Stanforth, R., Natarajan, V., Ledsam, J. R., MacWilliams, P., Kohli, P., Karthikesalingam, A., Kohl, S., Cemgil, T., Eslami, S. M. A., and Ronneberger, O. (2020) Kontrasttraining für eine verbesserte Off-of-Distribution-Erkennung. [Xu et al., 2020] Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. (2020). Recipes for safety in open-domain chatbots. . arXiv preprint arXiv:2010.07079 [Zellers et al., 2019] Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., und Choi, Y. (2019). Hellaswag: Kann eine Maschine wirklich Ihren Satz beenden? [Zhang et al., 2020] Zhang, H., Li, A., Guo, J., und Guo, Y. (2020). Hybridmodelle für offene Set-Erkennung. [Ziegler et al., 2019] Ziegler, D., Stiennon, N., Wu, J., Brown, T., Amodei, D., Radford, A., Christiano, P., and Irving, G. (2019). Fine-Tuning GPT-2 from Human Preferences. Dieses Dokument ist unter der Lizenz CC by 4.0 Deed (Attribution 4.0 International) verfügbar. Dieses Dokument ist unter der Lizenz CC by 4.0 Deed (Attribution 4.0 International) verfügbar.