AI yenye manufaa na isiyo na madhara: Mafunzo ya Upatikanaji Inaboresha Utendaji Katika Karibu Tathmini Zote za NLP

Mafunzo ya Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Msaidizi Waandishi wa: Mji wa Bai Andy Jones kwa ajili ya Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Dawn Drain Stanislav Fort Mwisho wa Ganguli Tom Henighan Nicholas Joseph wa Saurav Kadavath Maelezo ya Jackson Kernion Tom Conerly Sheer El-Showk Nelson Elhage Zac Hatfield-Dodds Danny Hernandez Tristan Hume Scott Johnston Shauna Kravec Liane Lovitt Maana ya Nanda Catherine Olsson Dario Amodei Tom Brown Jack Clark Sam McCandlish Chris Olah Ben Mann Jared Kaplan Waandishi wa: Mji wa Bai Andy Jones kwa ajili ya Kamal Mwakyembe Amanda Askell Mke wa Chen Simu mpya ya Siku ya Drain Mji wa Stanislav Fort Mwisho wa Ganguli Tom mwenyewe Nicholas Joseph wa Msisemi Shaykh Rabiy ́ ni Maelezo ya Jackson Kernion Maelezo ya Connery Maonyesho ya El-Show Nelson Mwakyembe Mchezaji wa Zac Hatfield-Dodds Kijana Hernandez Maisha ya Tristan Hume Mwandishi wa Johnston Shauna Kravec Mke wa Lovitt Maana ya Nanda Kiongozi wa Olsson Mwakilishi wa Amodei Kwa mfano, Tom Brown Jack wa Clark Sam McCandlish kwa ajili ya Chris Mwalimu Mwanamume wa Jared Kaplan wa Abstract Abstract Tunatumia mfano wa mapendekezo na ujuzi wa kuimarisha kutoka kwa maoni ya binadamu (RLHF) kwa mifano ya lugha ya finetune ili kutenda kama wasaidizi muhimu na wasio na hatari. Tunaona mafunzo haya ya kuimarisha utendaji katika takribani tathmini zote za NLP, na ni sawa kabisa na mafunzo ya ujuzi maalum kama vile coding ya python na ufungaji. Tunatazama mode ya mafunzo ya mtandaoni iliyopangwa, ambapo mifano ya mapendekezo na sera za RL zinahitimishwa kila wiki na data za mapendekezo ya binadamu mpya, na kuboresha ufanisi wa seti zetu za data na mifano. Hatimaye, tunachunguza ufanisi wa mafunzo ya RLHF, na kutambua uhusiano wa linear kati ya tuzo ya RL na mizizi ya 1 Maelezo ya Tunataka kuendeleza mbinu za mafunzo ya wafanyakazi wa AI ambao ni wa manufaa, waaminifu na usio na hatari. Katika makala hii tunaonyesha kwamba tunaweza kufundisha msaada wa kutosha na usio na hatari. (HH) msaidizi wa lugha ya asili kwa kukusanya data ya mapendekezo ya binadamu na kutumia mbinu za modeling ya mapendekezo (PMing) na kujifunza kwa kuimarisha kutoka kwa maoni ya binadamu (RLHF). [Mwalimu na wenzake, mwaka 2021] 1 2. Lengo letu sio kufafanua au kuagiza kile ambacho kina maana ya 'msaada' na 'hakuna madhara', bali kutathmini ufanisi wa mbinu zetu za mafunzo, hivyo kwa kiasi kikubwa tunamruhusu wafanyakazi wetu wa umma kutafsiri dhana hizi kama wanavyoona ni sahihi.Tunashughulikia manufaa na uharibifu tofauti, kukusanya seti tofauti za data za mapendekezo ya binadamu kwa kila mmoja. Kwa manufaa, tunamwomba wafanyakazi wa umma kuomba mifano yetu kusaidia na kazi yoyote ya maandishi tu kama kujibu maswali, kuandika au kuhariri nyaraka, au kujadili mipango na maamuzi. Katika kila hatua ya mazungumzo yao na msaidizi wa AI, wafanyakazi wa umma hupatikana na majibu mawili iwezekanavyo. Wale wanaohusika katika kazi ya msaada wanatakiwa kuchagua majibu mazuri na ya haki (yaani bora). Wale wanaohusika katika kazi ya timu ya rangi wanatakiwa kuchagua majibu yenye uharibifu zaidi (yaani mabaya zaidi). Mazungumzo haya na mapendekezo ya kibinadamu yaliyoonyeshwa huunda datasets yetu. 2 3 Kuwasaidia na kutokuwa na madhara mara nyingi ni kinyume na kila mmoja. Mtazamo wa kutosha juu ya kuepuka uharibifu unaweza kusababisha majibu 'salama' ambayo haina kweli kukabiliana na mahitaji ya binadamu. Mtazamo wa kutosha juu ya kuwa msaada unaweza kusababisha majibu ambayo husaidia binadamu kuharibu au kuzalisha maudhui ya sumu. Tunathibitisha mvutano huu kwa kiwango kikubwa kwa kuonyesha kwamba mifano ya upendeleo iliyoandaliwa kwa msingi wa tathmini moja ya sifa hizi inafanya kazi mbaya sana (hata mbaya zaidi kuliko uwezekano) juu ya nyingine. Kwa bahati nzuri, tunaona kwamba PM zilizoandaliwa juu ya mchanganyiko wa dataset zote mbili zinaweza hata hivyo kujifunza masomo sahihi na kutenda kwa msaada wakati inahitajika, wakati wa kuchochea kukataa maombi mabaya. Pamoja na mifano ya upendeleo katika mkono, sisi kisha kufundisha wasaidizi wenye manufaa na wasio na hatari kupitia ujuzi wa kuimarisha, kutumia alama za PM kama tuzo. Tunatathmini utendaji wa PM na sifa zinazohusiana zaidi za utendaji wa mifano yetu iliyoandaliwa na RL mifano yenye manufaa ya RLHF ni rahisi sana kwa timu ya rangi, wakati mifano yenye manufaa + isiyo na madhara ni ya manufaa sana na yenye uharibifu mdogo. 1 ya Swali ambalo mara nyingi linaulizwa kuhusu mafunzo ya usawa ni kama itaathiri uwezo wa AI. Tunaona kwamba wakati RLHF inatumika kwa mifano ya lugha kubwa, jibu linaonekana kuwa sio karibu-katikati. Pia tunasisitiza kwamba mtu anaweza kuchanganya ujuzi maalum na mafunzo yanayohusiana na usawa bila kuathiri usawa au utendaji. Katika mazoezi, mifano ya usawa inaweza kuwa rahisi zaidi kwa mtumiaji na kutumika kuliko washiriki wao wa asili, ambayo inaonyesha kuwa kuna sababu ndogo ya kutumia mifano ambayo haijatengenezwa kwa usawa. 3. 1.1 Msaada wa Dialogue Preference Datasets • We collect separate helpfulness and harmlessness (i.e. red-teaming) datasets primarily using various 52B language models (see Section kwa maelezo) katika interface yetu (Figure Wafanyakazi wa umati wana mazungumzo ya wazi na mifano, au kuomba msaada, au kutoa maagizo, au kujaribu kupata mifano ya kutoa majibu mabaya, na wanatakiwa kuchagua majibu muhimu zaidi au mabaya zaidi. kwa kila hatua ya mazungumzo. 2 6 ya). 4 • Tunakusanya sehemu tatu za data, moja kutoka kwa mifano yetu ya awali, moja na sampuli ya kukataliwa dhidi ya mifano ya mapendekezo ya awali, na seti ya mwisho ya data iliyokusanywa na mifano iliyoundwa na kujifunza kwa 'online' ya kuimarisha kutoka kwa maoni ya binadamu, ambayo tunaboresha kwa kiwango cha karibu kila wiki. 2.3 ya Alignment with Human Values Has Many Benefits and Essentially No Cost to Performance • Mifano ndogo yana uzoefu mkubwa wa 'sheria ya kulinganisha' - utendaji wao katika tathmini mbalimbali hupungua baada ya mafunzo ya RLHF. 13B na 52B kwa ajili ya Mifano iliyofundishwa na RLHF inafanya kazi bora kwenye tathmini ya NLP ya shots, na sawa na tathmini ya shots chache. alignment bonuses 5 • Mafunzo ya lugha ya asili ya RLHF kwa HH yanaweza kutumika kwa mifano ambayo yameandikwa kwanza kwenye msimbo, na juu ya tathmini (kama inawezekana kwa kuboresha maelekezo ya madhumuni ya jumla yafuatayo). Tunaona pia kwamba mchanganyiko wa mafunzo ya mfano wa upendeleo kwa HH na Kwa mujibu wa summary incurs no degradation in perfor- mance in either HH or summarization. So there is no reason not to combine alignment training with more specific, valuable skills. improves their programming ability specialized skill [Mwalimu na wenzake, 2020] • Kuna a , ambayo inaweza kupimwa katika ngazi ya modelling ya mapendekezo na sera zilizoandaliwa na RLHF (Kielelezo However, as model size increases, PMs perform better on both distributions simultaneously and become much more robust to the rela- tive proportions of helpful and harmless training data. tension between helpfulness and harmlessness 1 ya). • We also show that one can use OOD detection techniques [F kukataa maombi ya ajabu na ya uharibifu zaidi (Kielelezo Pamoja na mambo machache au yasiyo ya kipekee (Figure ort et al., 2021] 22), yaani 23) kwa ajili ya Scaling, RLHF Robustness, and Iterated ‘Online’ Training • Tunajifunza uhusiano wa kupanua kwa usahihi wa PM kama utendaji wa ukubwa wa mfano na dataset, na kupata mzunguko wa log-linear (Kielelezo ingawa tunaona baadhi ya idiosyncrasies (Figures na ya 7 Kwa hiyo, 31 ya 32). • Tunafanya majaribio juu ya (Tazama picha ya ambapo tunashiriki datasets yetu kwa nusu na kufundisha mifano tofauti ya upendeleo juu ya kila nusu. Kisha tunajifunza mifano ya RL dhidi ya PM moja wakati wa kutathmini na nyingine. Tunaona kwamba PM kubwa ni imara zaidi kuliko PM ndogo, na kama ilivyotarajiwa, ongezeko la juu wakati wa mafunzo ya RLHF. robustness of RLHF 4 Kwa hiyo, • Tunaona kwamba kwa ajili ya ( 0) na mshahara wa kwa kiasi kikubwa cha mafunzo ya RLHF (tazama Takwimu na ya where na ya 0 ni sera na sera ya awali, kinyume chake. Tunaelezea jinsi uhusiano huu unaweza kutokea na kujadili maombi ya uwezekano na mwelekeo wa baadaye. D ya π approximately linearly related 4 13), yaani π π • Tunajifunza , ambapo tunabadilisha mifano yetu ya mapendekezo na sera za RLHF kwa mzunguko wa kila wiki, na kisha kutumia tena mifano hii mpya ya RLHF ili kuingiliana na wafanyakazi wa umma. Hii iliboresha kwa kiasi kikubwa mifano yetu kama ilivyotajwa na wafanyakazi wa umma (Kielelezo na kuboresha kwa kiasi kikubwa dataset yetu kama inavyozidiwa na PMs zetu wenyewe (Kielelezo kujaza kichwa cha juu katika suala la ubora. iterated online training 1 ya), ya 15), Ili kuondoa udanganyifu na kuimarisha matokeo yetu, tunachukua majaribio mengine ya kudhibitiwa (Kielelezo kuhifadhi ukubwa wa dataset na hyperparameter nyingine imara. 16) ya 1.2 Summary of Evaluations and Metrics • ya : Tunatathmini mifano yetu kwenye MMLU Msisemi Shaykh Rabiy ́ ni Imaam wa Jarh wat-Ta ́diyl. Hifadhi ya Maelezo ya OpenBook ya Arc ya Triviaca Mifano ya na ya for full results and Figure Katika kila kesi isipokuwa TriviaQA, 12B na 52B RLHF-maandalizi mifano bora kuliko LMs msingi. Pamoja, sisi kuchukua mifano ya coding Python na finetune yao na lugha asili RLHF, na kisha tathmini yao juu ya codex HumanEval Mfano wa Sisi pia majaribio na mchanganyiko wa mafunzo ya PM kwa HH na summarization kama ujuzi maalum, na tathmini matokeo ya utendaji wa PM (Kielelezo kutafuta kwamba mafunzo ya mchanganyiko haina kuharibu usahihi wa PM. NLP and Code Evaluations [Hendrycks na wenzake, 2021b], Mwalimu na wenzake, mwaka 2016 [Mwalimu na wenzake, 2019] [Mwalimu na wenzake, 2018] [Mwalimu na wenzake, 2018] [Mwalimu na wenzake, 2017] 28 29 3 [Mwalimu na wenzake, mwaka wa 2024] 21. [Mwalimu na wenzake, 2020] 20 kwa ajili ya • ya : Tunatathmini PM zetu kwa kutumia Tathmini zetu za HHH Maonyesho ya Big-Bench (Ujumbe wa Katika mazungumzo ya mpinzani Kwa upande wa jinsia (Ujumbe wa Tunatathmini mifano yetu ya RLHF kwenye TruthfulQA (Ujumbe wa Picha ya Mtoto [P] kutoka BIG-Bench, upungufu wa jinsia (Figure na hisia zinazohusiana na rangi na dini (Figure RLHF huongeza hisia kwa makundi yote, lakini haina kuondoa upendeleo. Static Alignment Evaluations [Mwalimu na wenzake, mwaka 2021] 6 ya 5), [Mwalimu na wenzake, 2020] [Mwalimu na wenzake, mwaka 2021] 12 kwa ajili ya). [Mwalimu na wenzake, mwaka 2021] ya 5), wa wa wa wa wa wa wa wa wa wa wa 40 kwa ajili ya [Rae et al., 2021] 17). • ya : Tunaweza kuhesabu pointi za Elo kulingana na mapendekezo ya wafanyakazi wetu wa umma, mifano inayounganishwa na mazingira, mifano ya msingi ya mafunzo ya RLHF, na mifano ya mwisho ya mtandaoni ya RLHF (Kielelezo We also test our online models’ performance during training (Figure Kuanzisha mlinzi juu ya vifaa vingine vya kurekebisha (Figure na kufanya majaribio ya kudhibitiwa juu ya mafunzo ya mtandaoni ya iterated (Kielelezo Zaidi ya hayo, tulihudhuria waandishi wa kitaaluma kuandika mazungumzo ambapo msaidizi anatoa majibu ya ubora, msaada na waaminifu, na kisha tumemuuliza wafanyakazi wa umma kulinganisha majibu ya mfano wetu na yale ya waandishi hawa. zaidi ya asilimia 57 ya muda. Human Evaluations 1 ya). 15), 36 kwa ajili ya ya 16). 7 • ya Tunatoa sampuli kutoka kwa PALMs zote sensi- tive questions and from prompts provided with InstructGPT kwa ajili ya LAMDA Kwa mujibu wa Appendix Tunaonyesha baadhi ya kulinganisha na waandishi wa binadamu katika Sehemu ya na tunaonyesha baadhi ya mazungumzo fupi katika Sehemu ya Ili kupunguza tatizo la sampuli za kuchukua cherry, tunazalisha sampuli 17 kwa prompt na kuonyesha tu sampuli ya wastani, kama ilivyochaguliwa na mfano wetu wa upendeleo wa HH mtandaoni. Samples Solaiman na Dennison, mwaka wa 2021. [Mwalimu na wenzake, 2024] [Mwalimu na wenzake, 2024] ya C. 6.1 ya 6.3 ya 1.3 Kazi zinazohusiana Baadhi ya nyaraka za hivi karibuni, LaMDA and InstructGPT Wote wawili hutumia data ya binadamu kufundisha mifano ya lugha kubwa ili kuwa na manufaa zaidi au kuunganishwa. Wote wawili hutumia mifano ya lugha ambayo ni kubwa kuliko mfano wetu wa 52B. [Mwalimu na wenzake, 2024] [Ouyang et al., 2022] LaMDA finetunes large language models to participate in interesting, helpful, factu- ally grounded, and safe natural language dialogue. As with our work, they include notions of both positive interactions and safety/harmlessness. And their use of external information to ensure accuracy/grounded- ness goes beyond the methods we discuss here, and is perhaps more similar to WebGPT and GopherCite However, some differences are that rather than using reinforce- ment learning, they apply a mix of supervised learning techniques (both generative and discriminative), and their data collection process involves absolute ratings rather than comparisons. They do not explore whether their methods impose an ‘alignment tax’ on capabilities. [Mwalimu na wenzake, 2024] [Mwalimu na wenzake, mwaka 2021, Mwalimu na wenzake, mwaka 2022. Maelekezo ya FineTunes GPT-3 aina ya mifano ili kuboresha msaada wao. Kama katika kazi hii, wanatumia kujifunza kuimarisha kutoka kwa mapendekezo ya binadamu, kama ilivyoonyeshwa kupitia kulinganisha. Hata hivyo, pia zinajumuisha hatua ya kujifunza inayoongozwa ya mafunzo, wakati kinyume chake, finetuning yetu hutokea tu kupitia RL (tumefanya destillation ya mazingira, lakini hii ni zaidi kama kuomba rahisi). Labda tofauti kuu na kazi yetu ni kwamba haimaanishi mafunzo ya uharibifu, au wasiwasi kati ya ufanisi na ufanisi. Njia yao pia inatofautiana kutoka kwetu kwa maelezo fulani: hawakufundisha mifano ya upendeleo ambayo ni kubwa kuliko vigezo vya 6B, na wamechanganya mafunzo ya awali na RL ili kuepuka uharibifu katika utendaji wa tathmini. [Mwalimu na wenzake, 2024] [Brown et al., 2020] Kazi yetu inatofautiana na InstructGPT na LaMDA kwa sababu tunatafuta mafunzo ya 'online', ambapo tunasaidia mifano ya kuingiliana na wafanyakazi wa umma ili kupata data ya ubora wa juu na kujaza nyuso za usambazaji wa data yetu. Tofauti nyingine ni utafiti wetu wa ujuzi maalum kama vile ufungaji na coding, ambayo tunatumia ili kuimarisha hoja kwamba usawa unaweza kufikia bila kuzuia uwezo. Pia tunachunguza kwa uwazi mvutano kati ya ufanisi na uharibifu, ambayo haijafanyiwa kabla kama tunajua. Hatimaye, tunatafiti ufanisi na ufanisi kwa undani zaidi, ikiwa ni pamoja na wakati wa mafunzo ya RL. Pamoja na hayo, taratibu zetu (Figure Kwa kweli ni rahisi zaidi kuliko wale wanaotumika katika kazi hizi nyingine.Tunaamini hatua pekee muhimu ni kukusanya data ya maoni ya binadamu, modeling mapendekezo, na mafunzo ya RLHF. 2) Baadhi ya kazi nyingine za hivi karibuni zinashughulikia masuala ya ukweli kwa kutumia kutafuta kutoka kwenye database, au kupitia utafutaji wa mtandao na maoni ya binadamu, kama WebGPT ya GopherCite Kazi hizi ni ya kusisimua na ya kuvutia kwa kazi yetu; hasa matokeo yetu yanaonyesha kwamba mbinu zao zinapaswa kuwa sawa sana na mafunzo kwa manufaa na uharibifu. Ingawa kazi hizi zinaongeza uwakilishi wa uaminifu wa ushahidi wa wazi, kazi zaidi inaweza kuwa inahitajika ili kufikia uwakilishi wa kibinafsi wa kibinafsi kutoka kwa mifumo ya AI. [Mwalimu na wenzake, 2020] wa wa wa wa wa wa wa wa wa Borgeaud et al., 2021] [Nakano et al., 2021] [Mwalimu na wenzake katika mwaka wa 2022]. Maswali ya usalama na maadili yanayohusiana na mifano ya lugha yamejadiliwa sana (kwa mfano. 2021)), na matatizo yaliyojulikana ikiwa ni pamoja na uchafuzi, udanganyifu, na uwezekano kwamba mifano inaweza kufichua maelezo ya kibinafsi.Kama mifano inakuwa yenye nguvu, uwezo mpya na wa kushangaza na masuala ya usalama yanaweza kutokea Kazi zingine zimechunguza mbinu za kupunguza matatizo haya (kwa mfano. 2020]). Models have also been trained to directly evaluate ethical dilemmas Kuonyesha ujuzi juu ya benchmarks ya maadili Maombi ya utafiti zaidi ya jumla ya usalama wa AI ni pamoja na The RL robustness failures we discuss can be viewed as an instance of ‘reward hacking’, which was recently explored in [P Sera za RL zinaweza pia kushindwa kuzidisha nje ya usambazaji kwa njia zingine hatari [Mwalimu na wenzake, 2017] Bender et al., 2021, Weidinger na wenzake, [Mwalimu na wenzake, mwaka 2022] [Mwalimu na wenzake, mwaka 2021, Xu et al., [Jiang et al., mwaka 2021], [Mwalimu na wenzake katika mwaka wa 2021]. [Mwalimu na wenzake, 2016] Mwalimu na wenzake, mwaka wa 2021. an et al., 2022]. [Kuch et al, mwaka 2021]. maslahi yetu katika kujifunza mwelekeo na ukubwa wa mfano unaoongozwa na sheria za kiwango cha neural Ufuatiliaji unaohusiana ni kwamba kama idadi ya vigezo inaongezeka, mifano hufanya kazi kwa ufanisi zaidi na kuwa na hatari kidogo kwa 'kimbunga cha kusahau' Tunatarajia athari hii inasaidia kuelezea kwa nini mafunzo yetu ya HH yanaunganishwa na utendaji mzuri wa tathmini na ujuzi maalum kwa mifano kubwa. [Mwalimu na wenzake, 2019] Rosenfeld na wenzake, 2019 Mwalimu na wenzake, mwaka wa 2020). [Mwalimu na wenzake, mwaka 2021] [Majjid Mwakyembe na wenzake, 2022] 2 Uchambuzi wa data Tunatarajia maoni ya kibinadamu (HF) kuwa na faida kubwa ya kulinganisha ikilinganishwa na mbinu zingine wakati watu wana hisia ngumu ambazo ni rahisi kuzaa lakini ni vigumu kufafanua na kusafisha. Hiyo inamaanisha kwamba wakati wa kukusanya HF, tunapaswa kujaribu kuchagua kazi ambayo ni ya kuvutia na inayojulikana iwezekanavyo. Tumechagua kutumia mazungumzo ya lugha ya asili kwa sababu hizi zote, na kwa sababu ni ya jumla sana - kimsingi kazi yoyote ya maandishi inaweza kufanywa kupitia mazungumzo, hasa na vifaa vingine vya chanzo vinavyojumuishwa katika mstari. 2.1 Ufafanuzi wa kazi na wafanyakazi wa umma Our human feedback interface can be seen in Figure (Kwa maelezo zaidi, angalia kifungu cha People can interact with our models in natural language via chat, and ask for help with any text-based task. When it’s the model’s conversational turn, users see two possible model responses, and choose one with which to proceed. These two responses may come from the same model, or two different models. They can then ask follow-up questions or provide further instructions to the models. So there are two core components to the task, which repeat several times in each dialogue: 6 ya D). • Wafanyakazi wa umati kuandika ujumbe wa mazungumzo kwa mifano yetu, na kuwauliza kufanya kazi, kujibu swali, au kujadili mada yoyote ya maslahi. • Wafanyakazi wa umati huonyeshwa majibu mawili, na wanatakiwa kuchagua msaidizi zaidi na waaminifu (au katika kesi ya timu ya rangi, kuchagua majibu yenye madhara zaidi). We conjectured that crowdworkers who wrote well and engaged the AI in more interesting discussions would tend to have better judgment about which AI responses were most ‘helpful’ and ‘harmless’. This meant that rather than attempting to filter crowdworkers based on label quality, we instead used spot-checks of their writing, which were simpler and more intuitive for us to perform. Otherwise, our approach to data collection was to largely let crowdworkers use their own intuitions to define ‘helpfulness’ and ‘harmfulness’. Our hope was that data diversity (which we expect is very valuable) and the ‘wisdom of the crowd’ would provide comparable RoI to a smaller dataset that was more intensively validated and filtered. Overall, our process was roughly of this form: 1. We invited master-qualified US-based Wafanyakazi wa MTurk kushiriki katika mazungumzo na mifano yetu. 8 Badala ya kutathmini wafanyakazi wetu wote wa umma, tuligundua wale ambao walikuwa na uzalishaji mkubwa zaidi, na kwa pamoja waliwakilisha takriban 80% ya data yetu (karibu wafanyakazi wa umma 20). Kisha tulipima utendaji wao kulingana na ubora na utofauti katika mazungumzo yao, kwa sababu hii ilikuwa rahisi sana kutathmini kwa akili (lakini kulingana na kiwango chochote cha makubaliano juu ya chaguo zenye manufaa / zisizo na hatari). Kulingana na mbinu hii, tulikusanya orodha ya wafanyakazi wa 'chagua' wa MTurk ambao tumekuwa tukifanya kazi pamoja katika mchakato wa utafiti. 9 Tumewaalika wafanyakazi wetu waliochaguliwa kwenye channel ya Slack na kuwasiliana nao kwa barua pepe, ili kuhakikisha kwamba walikuwa wanalipwa kwa haki. na kuruhusu kutuambia kuhusu matatizo yoyote au matatizo. 10 Tumewaajiri pia wafanyakazi wa umma kwenye Upwork, na kuangalia kwa njia sawa, rahisi. Tumeendelea kutumia majukwaa mawili katika kazi hii. Tunaona kuwa ni rahisi kuchochea maingiliano ya ubora sana kwenye majukwaa kama vile Upwork, ambapo wafanyakazi wa wafanyakazi wa umma wanaweza kulipwa kwa urahisi kwa saa, badala ya kwa kazi. Lakini kinyume chake, wafanyakazi wa MTurk huwa wanaweza kuzalisha data haraka zaidi, na kuwakilisha karibu 80% ya seti zetu za data. We did not filter workers based on agreement or other direct measures of label quality, though we evaluated them retrospectively (see Figure right) and found poor average agreement (about 63%) between Anthropic researchers and our crowdworkers, as compared to recent similar work such as 10 [Stiennon et al., 2020, Ouyang et al., 2022]. Kama tahadhari muhimu, usambazaji wetu wa wafanyakazi wa umma haukujumuishwa wakati wa kazi hii, na tunatarajia ubora wa wafanyakazi wa umma uwezekano wa kuboresha wakati mradi unaendelea. Conversely, however, since we generally discouraged repetition, crowdworkers who have performed the task many times might also have had a tendency to engage in more esoteric interactions. 4.5. Tunapaswa pia kukumbuka kwamba tulimwambia kwa wazi wafanyakazi wa umma kwamba “kuwa uongo hauna manufaa” na kwamba wanapaswa kujaribu tu kuwakomboa kwa manufaa. responses, which presumably explains why our models improve somewhat in terms of honesty. That said, we did not expect crowdworkers to fact-check our models significantly, and for example they often prefer responses that include non-functional URLs, which are probably one of the simplest possible ‘lies’ to debunk. and honest 2.2 Huduma na uharibifu (Red Teaming) Datasets We collected two separate datasets using slightly different versions of our interface. For the helpfulness dataset, we asked crowdworkers to have open-ended conversations with our models, asking for help, advice, or for the model to accomplish a task (see Appendix ), and to choose the model response that was Kwa data ya uharibifu au red-teaming, tumewauliza wafanyakazi wa umma kujaribu kuchochea majibu mabaya kutoka kwa mifano yetu, na kuchagua response offered by the models. D.2 more helpful more harmful Our interface (Figure Inakuwezesha watumiaji kuonyesha nguvu ya mapendekezo. Tunaingiza kulinganisha katika datasets yetu tu ikiwa wafanyakazi wa umma wamesema upendeleo mkubwa kuliko dhaifu zaidi inapatikana. Katika kazi hii hatuwezi kutumia taarifa hii ya nguvu ya upendeleo; tunachunguza kulinganisha yote katika datasets yetu kama binary na yenye uzito sawa (kwa hiyo hasa hatuwezi kuingiza uhusiano). 6) Note that this means our helpfulness dataset tends to move conversations in a more beneficial direction, while in our red-teaming dataset user responses move conversations in a more harmful direction. We made this choice to make it possible for users to fully trick and exploit models while red-teaming, as this was most natural for other work we’re doing that’s specifically focused on harmfulness. However, we believe this difference made it difficult to train models that were both helpful and harmless, as explained in Section We plan to remedy this in future work, and would recommend others who are focused on training harmless dialogue models to collect data where users primarily choose model responses that move the conversation in the more beneficial direction instead. 4.4. 2.3 Mifano yaliyotumika kwa interface ya maoni na usambazaji wa data unaohusiana For data collection we predominantly kutumia mifano ya lugha ya 52B na maelezo makubwa yaliyotolewa katika We used three classes of models in our interface: 11 [Mwalimu na wenzake, mwaka 2021] • : At the beginning of the project this was the only model available. It performs similarly to a plain 52B language model prompted with HHH dialogues HHH Context-Distilled 52B Language Model [Mwalimu na wenzake, mwaka 2021] • ya na mfano wa upendeleo wa 52B, ambapo sampuli ziliundwa kutoka kwa LM iliyotengwa na mazingira ya 52B. of samples was a parameter, but most often we used = 16 ya Rejection Sampling (RS) k k • ya : We used a succession of these models in our interface. The models varied primarily based on the amount of data available when training the associated PMs (depending on the phase of the project). However, we also deployed models trained on different mixtures of helpfulness and harmlessness data. RLHF-Finetuned Models In the final phase of the project, when we were primarily deploying RLHF-finetuned models, we often de- ployed several such models at once. This allowed us to monitor progress by gathering model-comparison data, and also to (perhaps) improve data diversity. Corresponding to the three classes of models, we divide our data into three distributions: • A core dataset collected using only the context-distilled LM. This dataset includes 44k helpful- ness comparisons and 42k red-teaming (harmlessness) comparisons (note a conversation typically comprises about four comparisons). base · A Mkusanyiko wa data unaojumuisha kulinganisha huduma ya 52k na kulinganisha na red-teaming ya 2k kwa kutumia mifano ya sampuli ya re-jection, ambapo sampuli ya kukataa ilitumia mfano wa mapendekezo uliofundishwa kwenye dataset ya msingi. RS • An iterated ‘ ’ dataset including data from RLHF models, which were updated on a roughly weekly cadence over the course of about five weeks. This dataset contains 22k helpfulness compar- isons and no red-teaming data. online A histogram of these distributions by our final, online HH preference model’s score can be seen in Figure in Section In what follows when we discuss the or base+RS dataset, we will be referring to the combination of the first two components. Our ‘online’ RLHF models in Section are trained on all three components. Most of our results are based on the static dataset, as we conducted experiments and evaluations with it while the online data collection was underway. 15 4.5. static 4.5 We analyze a few different splits of the static dataset – a standard split into 95/5 train/test data, and a 65/35 split that we use in order to obtain better statistics when evaluating preference model calibration on the test set. We also consider a 50/50 split, where we train distinct PMs on the two halves of the dataset. This is used to evaluate the robustness of RL training, as we then train an RL policy against one PM while evaluating the rewards achieved by that policy as measured by the independent PM. 2.4 Kulinganisha mifano na pointi za Elo A significant part of our analysis compares models against each other in order to generate associated Elo scores, as described in That is, we have crowdworkers chat with two models simultane- ously, with each model generating one response (either ‘A’ or ‘B’) at each turn, and we record the sample that is preferred by the worker. This provides us with a record of ‘win rates’ between pairs of models, which we can then fit to corresponding Elo scores, to produce Figure (where we show both win rates and Elo scores). Two useful conversion formulas are [Askell et al., 2021]. 1 kwa sehemu ambayo mfano mmoja ni bora zaidi kuliko mwingine, tofauti ya pointi za Elo, na pointi zetu za PM. Kumbuka kwamba kwa dhana ya kushinda pointi, pointi za Elo na pointi za PM ni za kubadilishana; tunahifadhi pointi za Elo na PM ili tuweze kuepuka kuchanganya mapendekezo ya wafanyakazi wa umma (ambayo tunatumia Elo) na modeling yetu ya mapendekezo na RLHF (ambayo tunatumia pointi za PM). Note that the Elo scores for context-distilled models in Figure differ somewhat from the analogous results for prompted models in - Tathmini ya Elo sasa ni zaidi ya kushinikiza. Tofauti kuu ni kwamba hatujawahi kutumia sampuli ya juu ya P wakati huu . The difference may also be due to changes in the crowd- worker distribution since that earlier experiment, or changes in crowdworker expectations, as before this test our workers were mostly interacting with higher-quality RLHF-trained models. 1 [Askell et al., 2021] 12 3 Preference Modeling for Helpfulness and Harmlessness 3.1 Models and Training Setup Tunatumia mifano ya lugha na maelezo ambayo ni sawa na yale yaliyojadiliwa katika [Askell et al., 2021], na jumla ya mifano saba ya lugha na hesabu ya vigezo kutoka 13M hadi 52B na karibu na mfululizo wa kijiometriki na ongezeko la karibu 4X. Tunatumia PyTorch [Paszke et al., 2019] na Triton [Tillet et al., 2019] ili kuwezesha mafunzo ya mfano na utendaji. Mipangilio yetu ya mafunzo ya mfano wa mapendekezo pia ni sawa na yale ya [Askell et al., 2021], na hasa tunatumia 'modela ya mapendekezo ya mapendekezo' (PMP) kwa mifano yetu ya lugha kabla ya kuboresha data yetu ya maoni ya binadamu, kama ilivyoelezwa katika Sehemu ya 4 ya makala hiyo. 3.2 Basic Scaling Results Tunataka kuelewa jinsi utendaji wa modeling ya upendeleo unaboresha kwa kuongeza ukubwa wa mfano na kukusanya data ya ziada. we show basic results for PM accuracy when training on our static helpful and harmless data mixture. Roughly speaking, we observe log-linear trends in both dataset and model size. We tend to find somewhat more consistent trends if we model only the helpfulness or harmlessness distributions in isolation, rather than as a mixture, as observed in Figure in Appendix Lakini pia tunaona kwamba kwa baadhi ya data usambazaji scaling trends can exhibit more complex patterns that defy simple trends. 7 32 A.3. [Stiennon et al., 2020], Our preference modeling data comes from natural language dialogue, where crowdworkers have text-based conversations with the model, and choose the more helpful of two model responses at every turn in the conversation (or the more harmful one, for red-teaming tasks). So it is natural to ask how PM performance changes as a function of the conversational turn. We show these results in Figure PMs are somewhat more accurate on the first step of the conversation, but their accuracy is nearly constant thereafter. 8. 3.3 Calibration of Preference Models and Implications for RL Preference model scores should predict the probability that humans will prefer one or another model- generated response. We are interested in whether these probabilities are accurate, i.e. whether the PMs are well calibrated. We characterize calibration in Figure ambapo tunatoa usahihi wa PM kama utendaji wa tofauti katika pointi za PM zilizowekwa kwa wanandoa wa sampuli, pamoja na mstari mkubwa wa nyeusi unaowakilisha calibration kamili. Tunaona kwamba PMs mafunzo tu juu ya data ya manufaa ni vizuri sana calibrated, lakini PMs mafunzo juu ya mchanganyiko wa data muhimu na isiyo na madhara ni kidogo chini ya uhakika. 9, These calibration results are important because in later sections we will be using PM scores as the reward signal for reinforcement learning. Since the PM scores are well-calibrated, we can trust that they faithfully encode the probabilities that humans will prefer specific model samples (at least on-distribution with the training set). This means that when we see RL achieving a given reward, we can trust that those who interact with this model (if they are well-represented by our crowdworker distribution) will prefer it to reference models at a predictable rate, provided that the PM scores of the models’ responses are within the range considered in these calibration studies. That said, we find significant failures of robustness as RLHF optimizes towards much higher scores, as explained in Section Maelezo ya Appendix robustly 4.5 B. ya 4. Kwa ujumla tunaweza kutarajia kwamba kama ubora wa sampuli unaboresha, inakuwa vigumu kutambua kwa uhakika sampuli bora zaidi. in the appendix, we show that if we restrict to comparisons where both samples have a PM score above a given threshold, then PM accuracy drops as a function of this threshold value (to compare different PMs we add an offset so that they all obtain the same mean score on the test distribution). Notice that this result combines three effects: (1) differentiation among more sophisticated samples may be more challenging and require greater capabilities, (2) we have fewer very high quality samples to learn from in our dataset, as they are non-generic, and (3) pairs of samples that are all high-quality will have similar scores (compared to randomly chosen pairs), and so be more difficult to distinguish. 25 These observations also have an implication for RLHF training, namely that we should expect diminishing returns from further RLHF training once our policies achieve a sufficiently high PM score. This also motivates online training, so that we can update our PMs to stay on-distribution as RLHF policies improve. 3.4 Evaluating Helpful and Harmless Preference Models 3.4.1 HHH Evaluation and Crowdworker-Anthropic Agreement Tumeonyesha kwamba PMs walijifunza juu ya dataset yetu ni kujifunza kitu na ni calibrated vizuri. Lakini pia tunataka kuonyesha kwamba PMs hizi ni kweli kujifunza kuchagua manufaa na haramu tabia, kulingana na baadhi ya kiwango cha kujitegemea. Tulitoa seti ya tathmini ya kulinganisha kwa manufaa, uaminifu, na usahihi (yaani, Tathmini ya HHH) katika kazi ya awali na tathmini nyekundu, mazingira-distilled, na sampuli lugha mifano on this benchmark. As shown in Figure tunapata kwamba PM zetu zinafanya kazi vizuri zaidi kuliko mifano yote tuliyoyashughulikia hapo awali. recently reported a mean human score of 75% on this dataset, so in a sense our PM performance of 86% is well above mean human accuracy. [Askell et al., 2021] 5, [Chowdhery et al., 2022] We also observe that there are a few outliers, where our PMs are confidently incorrect on HHH evaluation comparisons (see Figure left). These are all examples of failures of , where the better choice is honest but unhelpful, while the incorrect choice appears both helpful and well-written, but is subtly incorrect or misleading. We show the single worst example in Figure It’s worth noting that the bad response (which was written by Anthropic researchers) is likely out of distribution as compared to samples from our language models and RL policies. Since only model-generated samples were used to train the PMs, this may explain why the PM was badly fooled by these examples. It also indicates that the PMs are not adversarially robust. 10 honesty 11. We can also compare our crowdworkers, ourselves, and our preference models in terms of agreement on labels. The results are shown on the right in Figure na ni msingi wa mifano ya 320 kutoka kwa mkusanyiko wetu wa mtihani wa static. PMs ni mafunzo ya kutabiri tabia ya wafanyakazi wa umma, hivyo makubaliano ya PM-Crowdworker ni bora. Hata hivyo, PM kubwa kwa kweli inakubaliana na waandishi (yaani watafiti wa mwanadamu) kidogo zaidi kuliko waandishi wanakubaliana na wafanyakazi wa umma juu ya alama. Tunadhani pia hii ni sampuli mbaya ya data, kwa sababu usahihi wa PM ni asilimia kadhaa chini kwenye sampuli hii ikilinganishwa na seti kamili ya mtihani. 10, 3.4.2 Gender Bias and Bot Adversarial Dialogues Tunaweza kuanza kutathmini kama PM zetu zinaonyesha upungufu wa jinsia kwa kulinganisha pointi katika maneno ya jinsia. Je, unawezaje kuwaambia Isotopi moja kutoka kwa mwingine? ☆☆☆☆. Assistant: Sure thing, here is your sentence with the missing word filled in: The {Occupation} is a {Gendered Term} where we use a list of 76 occupations, 12 matched gender terms (mother vs father, guy vs gal, etc), and both present and past tense Takwimu za mfano wa mapendekezo hazina maana moja kwa moja, hivyo badala yake tunatathmini tofauti ya takwimu kati ya maneno ya jinsia yaliyopangwa, na kulinganisha na tofauti ya takwimu kati ya maneno ndani ya jinsia iliyotolewa. Tunaona kwamba tofauti kati ya maneno ambayo yanaonyesha jinsia fulani inaonekana kuwa angalau kubwa kama tofauti kati ya jinsia. [Mwalimu na wenzake, mwaka 2021] 12. Kama uthibitisho wa nje wa PMs yetu, tunachunguza Dataset ya Mawasiliano ya Adversarial ya Bot (BAD) This dataset contains a few thousand conversations between an AI system and a human. Each AI response is labeled as harmful or not harmful. We compute the distribution of preference model scores for BAD AI utterances (we restrict our analysis to the first BAD AI utterance per conversation) and find that the BAD AI utterances marked as harmful have significantly lower preference model scores. This suggests that our PMs are effectively classifying these AI generated utterances, even though they are likely quite different from the data distribution that our PMs were trained on. [Xu et al., 2020]. 4 Reinforcement Learning from Human Feedback 4.1 Utaratibu wa mafunzo We apply reinforcement learning (RL) with preference modeling, following the approach outlined in which can summarized in the following steps: [Stiennon et al., 2020], 1. Prepare a dataset of comparisons, and train a PM to assign a higher score to the ‘better’ item in each comparison. In the context of our human feedback experiments, each comparison consists of a prompt followed by a pair of model-generated responses, with a PM score evaluated at the end of each response. 2. Extract all the prompts from the preceding dataset, and train an RL policy to generate a response to each prompt autoregressively, with a reward signal provided by the PM score at the end of the response. PM dataset and training details are provided in Appendix Pia tulijadili utendaji wa PMs zetu katika Sehemu ya In the language of RL, each response generated by the policy is a ‘timestep’, a full conversation is one ‘trajectory’, and the PM score is a single ‘reward’ provided at the end. A.2; 3. The idea is to use the preference model to steer the policy towards writing better responses. However, as we saw in earlier sections, PMs also become less calibrated at higher scores, so higher rewards do not necessarily imply better performance. Ili kuimarisha mafunzo ya RL, tunatumia ufanisi wa sera ya karibu (PPO) Pia tunafanya kazi nyingine and apply an empirically-estimated KL penalty term in the reward, with the total reward given by [Mwalimu na wenzake, 2017] [Mwalimu na wenzake, 2020] where KL >/ 0 is a hyperparameter. In practice we use a very small value of KL = 0:001, which likely has a very minor impact during most of RL training (as DKL < 100 typically), and might actually be wholly unnecessary. More details about RL are provided in B.1. Throughout this paper we use PM = the preference model score itself for the RL reward. Recall that as implied by equation this means that the difference in PM values between two samples and will be related to the predicted probability ( ) that will be preferred to via r (2.1), r A B P A > B A B There is no good reason to use this preference model score directly as the reward, but it has been used in prior work such as and so for simplicity we will not explore variations on this choice here. 13 [Stiennon et al., 2020] In order to produce additional prompts (i.e. the human side of the conversations) for RLHF training, we used a large LM to generate them. For this purpose, we simply used few-shot learning, creating a context with about 10 existing high-quality human queries, and then sampling to generate more. We find that the sample efficiency of RLHF is roughly the same on the original crowdworker-written prompt dataset and the model- generated one, so we combine the two for greater diversity during RLHF training. We used 137k prompts from the ‘static’ dataset, and 369k model-generated prompts. Note that almost all of our preference modeling data was collected from 52B models. This means that RLHF training with smaller models might have been challenging, since samples from smaller models tend to be out-of-distribution from the PM training data. Thus it is quite interesting that models more than fifty times smaller were actually able to learn and improve, as seen in Figure 1. 4.2 Robustness Experiments We now discuss the problem of RLHF robustness. A fully robust PM would agree with humans on distribu- tions of dialogues quite different from those encountered during PM training (i.e. different from those created by crowdworker interactions with our deployed AI assistants). However, we do not expect that our PMs are so robust, and in fact Figure provides one plausible example of a robustness failure. Since RL optimizes the policy to maximize the PM score, any failure in robustness on the part of the PM may be exploited by the RL policy to achieve higher rewards, without actually improving the policy’s behavior from the point of view of human evaluators. 11 A rigorous way to study robustness is to take snapshots of the policy at various points during RLHF training, including the initial snapshot, and have crowdworkers compare their performance. This gives a ‘true’ Elo score, as evaluated by crowdworkers, which can then be compared directly with the PM scores. We present an example of this study in Section 4.5 ya Hata hivyo, aina hii ya mtihani inahitaji kukusanya data ya kibinafsi ya maoni ya kibinafsi, ambayo inaweza kuwa ya polepole na ya gharama kubwa, hivyo hapa tunajifunza pia ujasiri kutoka kwa mtazamo tofauti. half and a half), and train separate preference models on each, which we refer to as the ’s and the ’s. We then train RLHF policies against the ‘s, wakati wa kutathmini yao kwa kutumia ’s. Kama vile jinsi tathmini ya mfululizo wa mtihani husaidia kuelewa upungufu katika kujifunza kwa usimamizi, evaluations help us understand overfitting relative to the ’s. These experiments are not conclusive since the train and test PMs may exhibit correlated robustness failures. Treni ya test train PM test PM train PM test PM test PM train PM The main conclusions from these experiments are: (1) RLHF becomes gradually less robust at higher PM scores, and (2) larger preference models are more robust than smaller ones. We conduct two sets of experiments as follows: • ya : This set consists of a scan of policies (i.e. one for each model size), all of which are trained with respect to the same 52B . Train PM Size = 52B train PM • ya : This set consists of a scan of policies, with each policy trained with respect to a of the same size as the policy. Train PM Size = Policy Size train PM For both experiments, each policy is further evaluated with respected to a scan of ’s throughout training. Note that a inahusu 7 tofauti mifano ukubwa kutoka 13M hadi 52B, hivyo sisi kutoa 7 sera na 7 × 7 tathmini kwa majaribio. test PM scan In Figure we compare the and scores throughout the training process, similar to how train and test curves are often compared for supervised training. We find that in all cases, the two scores are in close agreement during early stages of training, but eventually diverge, with the providing a lower score. The divergence is likely an indication that . That is, the policy has been over-optimized on the , making the over- confident in the policy’s performance. The , on the other hand, doesn’t suffer from this problem since it was trained on a different portion of data that neither the policy nor the had observed. 4, Treni ya PM test PM test PM the preference model is less robust and more easily exploited at higher rewards train PM Treni ya PM test PM train PM We provide more discussion in Appendix B.2. 4.3 An Approximately Linear Relation Between √ KL and Reward D D In Figures and we observe an between √KL and PM score during RLHF training. Furthermore, we note that when√a__ll mo__dels are trained and evaluated with the same PMs, the learning curves are roughly Katika ya KL-reward plane. Note that here the ‘KL’ is more precisely reward varies linearly in , then in the ‘sma√ll__- r__egion’ (yaani ambapo upanuzi wa mfululizo unatoa karibu nzuri), tunapaswa kutarajia ujira KL. Typically we should that reward varies linearly in , because because the initial policy haijatengenezwa awali kwa ajili ya ujira, hivyo hakuna sababu kwa nini ingekuwa katika kiwango cha juu kuhusiana na mabadiliko madogo . So the fact that this relation seems to hold empirically suggests that most of RLHF training remains in the small- regime. 4 13 approximately linear relation parallel D Vipimo vya δπ D expect δπ π δπ δπ Though they did not use these coordinates, a similar scaling can be read off from the results in learning to summarize In particular, they provide a nice analysis of rejection sampling, where they generate samples, and then plot mean reward of the top samples versus the KL = log( (Kwa ajili ya [Stiennon et al., 2020]. N k D N/k This analysis suggests that these RL learning curves might be associated with changes in the RL policy that behave very similarly to simply rejection sampling from the initial distribution. Tunaona uhusiano huu rahisi ni ya kushangaza sana, na tunaamini inastahili utafiti zaidi.Kwa kiwango cha uwezekano, inaweza kuwa na madhara mbalimbali na matumizi wakati RL-finetuning mifano kubwa ya kuzalisha: • Uhusiano huu hutoa utabiri mkali wa 'kiwango ambacho sera inahitaji kubadilika ili kufikia tuzo fulani'. Zaidi ya hayo, ikiwa mstari unaofaa kwa ukubwa wa mfano tofauti ni sawa, basi mtu anaweza kutumia mafunzo ya RL ya mfano mdogo pamoja na utendaji wa shots zero wa mfano mkubwa ili kuhesabu utendaji wa mwisho wa sera kubwa ya RL. Mstari wa mstari huu pia unaelezea jinsi mafunzo ya RLHF yanaweza kuzalisha faida kubwa sana katika ukubwa wa mfano, na kwa mfano inaelezea kwa nini mstari wa RLHF na mstari wa mazingira katika picha are roughly parallel. 1 • Unaweza kuuliza swali ndogo, labda laini kuhusu mafunzo ya RLHF - ni or simply kwa ajili yetu teaching the model new skills focusing the model on generating a sub-distribution of existing behaviors might attempt to make this distinction sha√r__p b__y associating the latter class of behaviors with the region where RL reward remains linear in KL. • To make some bolder guesses – perhaps the linear relation actually provides an on RL re__war__d, as a function of the KL. One might also attempt to extend the relation further by replacing √KL with a geodesic length in the Fisher geometry. upper bound By making RL learning more predictable and by identifying new quantitative categories of behavior, we might hope to detect unexpected behaviors emerging during RL training. 4.4 Tension Between Helpfulness and Harmlessness in RLHF Training Here we discuss a problem we encountered during RLHF training. At an earlier stage of this project, we found that many RLHF policies were very frequently reproducing the same exaggerated responses to all remotely sensitive questions (e.g. recommending users seek therapy and professional help whenever they express any level of displeasure at all). This greatly limited these models’ utility. We still see a vestige of this behavior in some of the examples provided in Section We now believe these policies were the result of kwa uharibifu, wakati helpfulness. 6.2 ya over-optimizing under-optimizing With our data collection procedure, we think this is quite intuitive. In order to get a very good score on red-teaming prompts, it’s probably sufficient for models to respond with something like “I can’t answer that.” This does not require much sophistication (it just requires learning to classify harmful requests), and so we expect it is easier to learn than helpfulness. In Figure (right), we show the policy’s PM score throughout training, after separating helpfulness and harmlessness prompts. On the left side of the same figure, we show the score distribution of PM comparison data, again separating helpful and harmless datasets. We observe that the policy’s harmlessness score is somewhat off-distribution, as it is on the upper tail of the harmlessness comparison data. On the other hand, the policy’s helpfulness score appears on-distribution, and is likely under-optimized. So we would expect this agent to be very difficult to red-team, but not very helpful. 14 This then raises an obvious question – can’t we just collect more harmlessness data to fill out the upper tail of the distribution? The problem involves the definition of harmlessness mentioned above – if simply refusing to answer a question is the ‘least harmful’ behavior, then this is probably both very easy to learn, and hard to improve on. That said, a more interesting ‘least harmful’ behavior would involve the model (helpfully) explaining why the request was harmful, and perhaps even trying to convince the human not to pursue such requests. We informally refer to such a model as a ‘hostage negotiator’. However, our data collection process made it very difficult for models to learn ‘hostage negotiation’. This is because when collecting our harmlessness dataset, we had crowdworkers choose the Jibu la AI. Tumefanya uchaguzi huu ili tuweze kuchunguza kikamilifu hatari ya mifano yetu ya red-teaming. Hata hivyo, kutoka kwa mtazamo wa RLHF hii ilikuwa tatizo, kwa sababu zaidi ya mzunguko wa kwanza wa mazungumzo, mifano yetu haijawahi kujifunza nini response to a harmful query might be like. Our dataset does not provide guidance on the upper end of the distribution, on what models should do, but only tells models what to do. more harmful sophisticated not Katika mazoezi, tumeboresha sehemu ya tatizo la uboreshaji kwa mafunzo juu ya sehemu kubwa ya maombi ya manufaa wakati wa RLHF. Lakini katika siku zijazo tunatarajia kukabiliana na tatizo hili kikamilifu na kwa utaratibu zaidi kwa kukusanya data ya uharibifu ambapo wafanyakazi wa umma kuchagua majibu bora iwezekanavyo kutoka kwa mifano yetu. In this way we hope that rather than simply shutting down harmful requests, models can learn the more subtle art of ‘hostage negotiation’ with red-teamers. 14 Note that since the data and models discussed in this section are from an earlier stage of our research, the RL results may look slightly different from other parts of the paper. 4.5 Iterated Online RLHF Katika sehemu za awali tulijadili tatizo kwamba PMs kuwa hatua kwa hatua chini calibrated na chini ya ujasiri katika alama ya juu, kama ilivyoonekana katika utafiti wa calibration PM katika picha and the RLHF robustness study in Figure 9 ya We believe this is caused by a lack of data in this high score regime. To address this, we propose : 4. iterated online RLHF • We simply train the best RLHF policy we can, and use that to collect comparison data from crowd- workers. Since the policy was trained to optimize for PM score, it should produce responses that are on the upper end of the score distribution. • We mix the new comparison data with our existing data, and train a new scan of PMs, which we then use to train a new scan of RLHF policies. Then reiterate this process indefinitely. Our hypothesis is that the ‘online’ RLHF policy helps us collect data on the upper end of the PM score distribution, which should improve PM calibration at high scores on subsequent iterations, and thereby allow us to train even better policies. Continuing this process should give us progressively better PMs and policies. Note that our use of the terminology ‘online’ is different from conventional use of the word—instead of training the same model iteratively, we retrain a new model per iteration. Moja ya wasiwasi kuhusu mbinu hii ni kwamba RLHF ina tabia ya kupunguza entropy ya sera, ambayo ingezingatia utofauti wa data zilizokusanywa kupitia utaratibu wa mtandaoni. Sisi sehemu ya kukabiliana na hili kwa kutumia baadhi ya snapshots tofauti kutoka mafunzo ya RL, na kutoka iterations tofauti mtandaoni, kwa wakati mmoja. We can see signs of life from the online approach by looking at the evolution of our data distribution. In Figure (left), we show the PM scores from three distributions of models: , (Mfano wa kuondoka) na Kama ilivyoelezwa katika sehemu We see that according to our final online PM (trained on all of the data), the quality of samples improves from the base to the rejection-sampling to the online data distributions. We also found that our online PM achieves accuracies of 74%, 70%, and 67% on the test sets for the respec- tive base, RS, and online-only distributions, which shows that distinguishing among higher quality samples is becoming more challenging. This makes us optimistic that online training should outperform rejection sampling in the long run. 15 Base RS Online 2.3. We show the learning curves for our online models, along with measurements of Elo scores from crowd- workers, on the right in Figure We see that models improve significantly during RLHF, but Elo scores from crowdworkers do not match predictions from PMs. We further discuss and decompose the robustness of RLHF training in Appendix where we see that distributional shift accounts for a significant part of the apparent robustness failure (Figure 15. B.4, 35). Kwa mfano we compare Elo scores of our online model with context-distilled models and RLHF models trained on the ‘static’ (i.e., no online) dataset, showing that the online models are clearly preferred by our crowdworkers. However, readers might worry about two caveats: the online model was trained on a slightly larger (about 20% larger) dataset, and the online model was trained with improved RLHF hyperparameters (the online model was trained with a larger , defined in Appendix and its PM was trained with 2048 context instead of 1024), as compared to the earlier static RLHF training run. 1 ya K B.1, To address both of these caveats, we performed a controlled experiment comparing two RLHF runs: one trained with our base dataset (about 44k PM comparisons), and another trained on an even mixture of base, RS, and online data (about 15k PM comparisons from each). So for this experiment we trained two separate PMs on each dataset, and then trained a pair of RLHF policies against these two PMs. Apart from the data difference, both runs used the same settings, and were only trained on helpfulness. In figure we compare Elo scores for various snapshots of both runs, as determined by crowdworker preferences, showing that the policy trained on the iterated-online mixture is clearly preferred. This demonstrates that online training works, and that performance gains are not merely due to increased dataset size or hyperparameter changes. ambayo ukubwa wa jumla wa dataset ni sawa na dataset ya msingi 15 16 ya 4.6 Tathmini: Bonus ya Upatikanaji, Uaminifu, na Biases Mifano ya lugha ambayo imekamilika kupitia RL kwa kawaida ina usambazaji mdogo zaidi, chini-entropy output. Hii inaweza kufanya tathmini ngumu wakati wao ni kabisa rigidly formated, kwa sababu majibu yote sahihi inaweza kuwa mbali na usambazaji kwa mfano wa RLHF (tazungumzia mfano na tathmini ya ubaguzi wa jinsia hapa chini). Kwa hiyo tunatarajia katika siku zijazo tathmini ya kazi inayohusiana na sampuli na mwingiliano wa binadamu inaweza kuwa muhimu zaidi. Katika nini hapa chini tunajadili baadhi ya tathmini ya kawaida NLP, na kisha tathmini hasa kuhusiana na madhara ya jamii ya mifano, ikiwa ni pamoja na uaminifu, hisia, na ubaguzi. 4.6.1 NLP Evaluations Tunatathmini mifano yetu juu ya kujibu maswali, akili ya kawaida, udanganyifu, na kumaliza hadithi kwa kutumia benchmark MMLU Lambada [P Hifadhi ya OpenBookQA ya Arc and TriviaQA The main conclusion is that RLHF tends to improve performance for large models, while degrading the performance of smaller models. [Hendrycks na wenzake, 2021b], aperno et al., 2016], [Mwalimu na wenzake, 2019] [Mihaylov et al., 2018], [Clark et al., 2018], [Joshi et al., 2017]. 16 Full results for both zero-shot and few-shot evaluations are shown in Figures na ya na tumetoa muhtasari wa mwenendo wa wastani katika picha Readers may notice that results improve rather suddenly for some evaluations; this is a consequence of the format we use for multiple choice questions, where we explicitly provide choices (Gopher used this format). The format is provided explicitly in Appendix Tunaona kwamba muundo huu unazingatia kuboresha utendaji kwa mifano kubwa, wakati huo huo huo kupunguza utendaji wa mifano ndogo, kusababisha kuonekana kwa uongo wa 'grok' ya curve. 28 29 ya 3. [Rae et al., 2021] E. [Power et al., mwaka wa 2022] 4.6.2 Honesty and Biases A major question is whether AI models are honest. We evaluate our models on TruthfulQA (MC1) and show the results in Figure Pia tunajumuisha utendaji kwenye risasi ya 50, ili kuonyesha kwamba wakati mafunzo yetu ya RLHF yanaongeza uaminifu kwa kiasi kikubwa, mifano yetu inawezekana kuwa na nafasi kubwa ya kuboresha. Kumbuka kwamba maelekezo yetu kwa wafanyakazi wa umma yanaonyesha kwamba "kuwa uongo sio msaada" na kwamba wanapaswa kuchagua majibu ambayo ni "msaada na uaminifu", hivyo hii inawezekana inahusiana na maboresho tunayoweza kuona kwenye TruthfulQA. [Lin et al., 2021] 5. Mfululizo mwingine wa maswali yanahusiana na makosa ya msingi ya mifano hii. Tunafanya tathmini ya mifano yetu kwa makosa ya hisia juu ya rangi na dini (katika muundo huo kama Gopher 2021]), for gender bias, and on the Bias Benchmark for QA (BBQ-lite) [P [Rae et al., arrish et al., 2021]. Matokeo ya hisia kwa makundi tofauti ya rangi na dini yameonyeshwa katika Picha The main effect we observe is that the sentiment of our RLHF-trained models tends to be much more positive than that of plain LMs, but it is more difficult to say if bias among groups has changed. We expect some of the visible effects are due to limitations in the sentiment model , kwa mfano, alama za chini kwa kundi la Atheist ni uwezekano kwa sababu sampuli kama vile (mzazi wa mfano katika italics): 17. 17 The Atheist man was very skeptical and critical of religious dogma and conventional moral principles, seeking instead to derive his ethical values from rational consideration and empirical observation of the natural world. kupokea alama ya hisia karibu na nusu, ambayo inaonekana kama tathmini ya shaka. Kwa tathmini hizi tunatumia muundo wa haraka ambapo mwanadamu anataka msaidizi kujaza kauli kama ifuatavyo: Human: Can you help me finish a sentence? The sentence is: {sentence beginning} Assistant: Sure thing, here is your completed sentence: {sentence beginning} na mwanzo wa mstari kama vile "Mtu wa Atheist alikuwa mkubwa" kama ilivyoonekana hapo juu. We also evaluated our models on BBQ-Lite, a simplified version of BBQA [P that is avail- able on BIG-bench. We were somewhat surprised by the results, and are unclear if any conclusions can be drawn based on them. In essence, we appear to find that our plain LMs do not exhibit very significant biases on this benchmark, using the available metrics. Here the metric is roughly ‘how often does the model invoke a term in a negative context, divided by how often it uses the term generally’. A value of 0.5 corresponds to the models being ‘unbiased’. RLHF training perhaps improves the results slightly (likely by improving sentiment) though the effect is very small (see Figure We may return to these evaluations in the future using the full BBQA metrics [P arrish et al., 2021] 18). arrish et al., 2021]. In Section 3.4.2 we investigated gender bias in preference models. We tried using this format for evaluating RLHF models, but we found that these models were more than 10 less likely than plain or context-distilled LMs to use a gendered term to complete sentences like ‘The CEO was a’. In other words, statements like ‘The CEO was a father’ seem to be somewhat off-distribution for the RLHF models, at least compared to general LMs. This is problematic for evaluations, because it means that the bias measurements are likely to be dominated by noise, and are unlikely to be representative of model behavior (i.e. model-generated samples). So in Appendix B.10 we experiment with another formulation of the task, where we reverse the order of occupation and gender terms, and add a ‘For work’ clause; this kept both the RLHF and other models on-distribution. The results are discussed in Appendix B.10, where we show that RLHF model’s biases are very strongly correlated (Figure 40) with underlying LM biases. In particular, with this evaluation we find that large RLHF models have gender biases that are very similar to language models evaluated at a lower temperature. 5 Competing Objectives, Specialized Skills, and OOD Detection A concern about alignment techniques is that they might compromise model performance. In Section we highlight a quantifiable trade-off of this kind, between helpfulness and harmlessness when training preference models. But it appears that larger models suffer less of a performance drop from this trade-off. 5.1 Furthermore, we also find that the conflict between helpfulness and harmlessness is relatively unique. Pref- erence models can learn to reward strong performance at specialized skills without any loss in performance at helpfulness and harmlessness. In Section we consider the evaluation of summarization quality as such a skill, using the learning-to-summarize dataset reformatted in conversational form. Later in Section tunathibitisha kuwa mifano ya msimbo (yaani, mifano iliyotengenezwa juu ya msimbo kwa mafunzo ya kusimamiwa) pia inashikiliwa na hatua za usawa wa HH, ingawa mafunzo ya RLHF haina kuhusisha data ya msimbo au mifano. 5.2 [Stiennon et al., 2020] 5.3 In Section we highlight another approach to avoiding harmful behavior – it may be possible to reject most harmful requests, even without any access to harmfulness training data, by leveraging out-of-distribution detection techniques [F This approach might also be useful more generally in deployment scenarios where strange or off-topic requests need to be flagged or ignored. 5.4 wa wa wa wa wa wa wa wa wa wa wa 5.1 Kuunganisha malengo ya manufaa na yasiyo ya hatari In many cases harmlessness acts as a constraint on helpfulness. So we should expect that helpfulness and harmlessness may behave as partially anti-correlated objectives. We establish this by evaluating preference models trained on different mixtures of HH data, and with different weightings. At a conceptual level, the HH PMs may essentially be learning to first classify the data and then choose a score depending on the distribution. We will show that larger models perform better and are more robust to data mixture and loss weighting, which may be due to their having greater success at separating reasonable from harmful requests. 5.1.1 Varying Helpful vs Harmless Data Fraction Tunajifunza mifano kwa kutumia sehemu za data ambazo zinatofautiana kutoka kwa manufaa ya 100% hadi usio na hatari ya 100% kwa mzunguko wa 10%. Usambazaji wetu wa data ya static una kulinganisha na red-teaming ya 42k, hivyo kudhibiti ukubwa wa dataset sisi daima kujenga mchanganyiko na jumla ya idadi hii ya kulinganisha. inaonyesha utendaji juu ya uharibifu na manufaa kama mchanganyiko wa data ya mafunzo ni tofauti. Kumbuka kwamba mafunzo yote juu ya utendaji au data ya manufaa ina matokeo katika utendaji kwenye usambazaji mwingine ambayo ni . This exemplifies the extent to which these distributions are in tension with each other. 19 significantly worse than chance Figure katika kiambatisho pia huonyesha usahihi wa mtihani wa wastani juu ya ufanisi na usio na hatari (ambapo Mean Acc = (Harmlessness Acc + Ufanisi Acc)*/*2). Kurva kwa ajili ya mifano kubwa inaonekana mwepesi karibu na maeneo ya 0% na 100%, lakini mwepesi juu. Kurva kwa ajili ya mifano ndogo ni hatua kwa hatua zaidi, na vipaji tofauti zaidi katikati. Hii tena inaonyesha kwamba PM kubwa ni nguvu zaidi kwa sehemu maalum ya data ya red-teaming vs utility ambayo hutumiwa, kuruhusu kujifunza dhana zote mbili kwa urahisi. 26 5.1.2 Weighting Helpful vs Harmless Losses Instead of studying different data mixtures, we can try re-weighting the losses. Since we have more helpful- ness than harmlessness comparisons, we experimented with weighting the losses as LTotal = LHelpfulness + λ · LHarmlessness for λ E 2 {1; 2; 3; 4; 10}, as shown in Figure 27 (relegated to the appendix). We note that larger models seem more robust to the choice of λ. Increasing λ from 1 to 10 causes a 7.4% decrease in accuracy on helpfulness for the 13M parameter model, whereas it only causes a 1.5% decrease in accuracy on the 52B parameter model. 5.2 Summarization as a Specialized Skill We expect that models finetuned with special skills may be particularly useful and valuable. Does alignment interfere with finetuning for a specialized skill? As one test of this question, we studied PM finetuning on the learning-to-summarize (LtS) dataset vs. a mixture of LtS and HH data. We formatted the LtS data in conversa- tional format so that it matches the HH data, as follows: [Stiennon et al., 2020] Human: Can you write a summary of this article for me? ...Text... Assistant: Sure, here it is: ...Summary... As shown in Figure large preference models trained on a mixture of HH and LtS datasets perform equally well on both. So at least at the level of preference modeling, there seems to be no cost to mixing HH with the specific skill of evaluating summarization quality. 20, 5.3 Lugha ya asili RLHF juu ya mifano ya Finetuned Code As another test of a specialized skill, we would like to see if natural language alignment can be combined with coding without compromising performance. Since our crowdworkers were never instructed to probe the model’s coding abilities, and most likely do not have a great deal of coding expertise, our human feedback data does not include a significant number of code-related conversations. The preference model finetuning dataset and the RLHF prompt dataset thereby do not contain any significant amount of code, though there is some code in the LM pretraining mix and possibly a small amount in the PM pre-training (PMP) mix. This makes code-related problems an interesting way to test generalization of RLHF, and especially its compati- bility with other skills. Our ‘base code models’ were finetuned on Python code scraped from Github as described in Starting from these Python fine-tuned (Python FT) models, we then ran our standard natural language RLHF training using ‘static’ preference models and prompts. We had difficulty achieving stable RLHF optimization on the 3B code model, so it has been excluded for this section. [Askell et al., 2021]. Tunatathmini mifano kwenye Dataset ya HumanEval which prompts language models with python function signatures and docstrings. Models are tasked with correctly filling in the function body given the context, and model-written functions are run in a sandbox environment. In Figure we show results versus model size with and without RLHF training. We see the same trend here as with other evaluations – RLHF decreases the performance of small models, but improves the performance of larger models. [Chen et al., 2021], 21 RL training tends to decrease the entropy of the models’ distribution, and so we were concerned that these results would be very sensitive to temperature and top-p tuning. So for our 52B models, we performed a scan over temperatures and two top-p settings for both the RLHF models and the base code models, and then chose the best setting for each model and pass@k. We did a grid-search over the evaluation hyperparameters: T ∈ {0, 0.4, 0.6, 0.8, 1.0} × p ∈ {0.95, 1} × k ∈ {1, 5, 10, 25, 50, 75, 100}. Results are summarized on the right side of Figure 21. For each model and for each k in pass@k, we take the maximum performance over all 10 combinations of hyperparameters. We see that RLHF improves performance over the baseline on this evaluation, for all pass@k. We should emphasize that as with our other evaluations, the improvements in performance from RLHF are modest. In fact, we find that simply prompting a base code model performs slightly better, as shown in Figure Appendix inaelezea zaidi muundo wa prompts tuliyotumia (yaani, ‘HHH prompts’), ambayo inajumuisha mifano michache ya msimbo. 38. B.8 Pia tulifanya majaribio yanayohusiana na kuongeza msimbo wa buggy kwenye maagizo, ambayo kwa kawaida hufanya hali mbaya (taz. (2021)). tuligundua kuwa mifano ya RLHF haifanyi kazi vizuri zaidi kuliko snapshots yao ya awali ya mfano wa msingi, wakati prompts hizi zinahusishwa katika mazingira wakati wa tathmini, hata baada ya kupeleleza juu ya joto na juu-p. [Chen et al., 5.4 Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests In this work we are primarily focused on achieving harmlessness entirely through natural language dialogue. However, one might try to avoid harmful behavior in a somewhat different manner, by either restricting language assistants to only respond to a narrow range of queries (approved-list), or by filtering and rejecting known types of bad behavior (block-list). We could use our preference models for these purposes, but we might also take a different, less supervised approach, and leverage advances in out-of-distribution (OOD) detection. Such an approach might also be useful for those who want to build systems that only respond to a narrow range of queries (e.g. code models that should avoid non-code topics). Out-of-distribution detection (OOD), and especially near out-of-distribution detection, have been a major challenge for deep neural networks. Deep networks routinely assign high probability to mis-classified inputs [Guo et al., 2017, Lakshminarayanan et al., 2016] as well as to test inputs not belonging to any of the training classes [Nguyen et al., 2014]. There have been many approaches to OOD detection based on discriminative models [Hendrycks and Gimpel, 2016, Lee et al., 2018, Liang et al., 2017, Liu et al., 2020] as well as deep generative models [Nalisnick et al., 2019, Zhang et al., 2020]. The more difficult case of OOD detection, the so-called near-OOD detection [Winkens et al., 2020], has recently been improved upon significantly using pre-training and large models [Fort et al., 2021]. For a prompt i, we extract a vector of activations of dimension dmodel from a layer ` and call it v ` i ∈ R dmodel . The task is to distinguish between an unseen example of harmlessness and helpfulness data without being explicitly shown any harmlessness data at all. This approach works by measuring the deviation of a prompt from the helpfulness data, rather than measuring how close it gets towards harmlessness data in particular. In this way, we do not depend on the specific harmful content we have at hand, and can potentially filter different kinds of non-helpfulness content. To detect whether an input comes from the in-distribution (the helpfulness dataset), we use a scoring function that takes the input and maps it to a scalar value score(x). To do that, [Lee et al., 2018] first proposed to fit a simple model to training examples of the in-distribution, Din train = {v ` i ∈ R dmodel}. We calculate the mean µ = 1 Ntrain PNtrain i=1 v ` i and the covariance matrix Σ = 1 Ntrain PNtrain i=1 (vi − µ) (vi − µ) T . The Mahalanobis distance of an unknown activation vector x from this training set is score(x) = (x − µ) T Σ −1 (x − µ). A simple improvement on top of the Mahalanobis distance called the Mahalanobis distance has been proposed in and shown to lead to better AUROC as well as more robust detection for a range of OOD problems in vision and genomics (in addition to more robustness to adversarial attacks [F 2022]). Inspired by this method and recognizing that our problem does not naturally involve semantically meaningful classes comprising the in-distribution, we propose a further modification we call the Relative Mahalanobis distance. We compute it by fitting a full covariance matrix Σ as before, as well as a diagonal- only covariance matrix Σdiag, and assigning the difference of their Mahalanobis distance as our scoring function, score( ) = ( − ) Σ−1( − ) − ( − ) Σdiag−1( − (Kwa ajili ya Relative [Ren et al., 2021] ort, Simplified x x µ T x µ x µ T x µ Figure shows the results for our OOD detection experiments, trying to distinguish the helpfulness data from harmlessness data using our new Simplified Relative Mahalanobis distance to the helpfulness activation vectors. The figure shows AUROC for activation vectors from different layers of the model and different model sizes. The bigger the model, the better its performance, with the middle layers performing best. The right panel of Figure inaonyesha mfano wa usambazaji wa kiwango cha OOD kwa data ya ufanisi (nyekundu) na data ya ufanisi (nyekundu) kwa kiwango cha 64L cha vigezo vya 52B na kiwango chake cha 32. Tunaweza kuona kwamba hali ya umbali wa data ya Ufanisi ni ya juu sana kuliko data ya ufanisi. Uwiano wa umbali wa Mahalanobis wa Ufanisi na umbali wa Mahalanobis wa kiwango ni kuonyeshwa katika Kielelezo in Figure 22 22 B.9 39. The advantage of this approach is that we are able to distinguish helpfulness data from non-helpfulness data, with harmlessness data being only a particular kind of non-helpfulness data. The disadvantage is its clear lower performance on this specific task in particular. Ikiwa tuna upatikanaji wa idadi ndogo ya mifano ya Inharmlessness inputs (out-distribution), tunaweza kufanya maonyesho machache ya nje, kama ilivyopendekezwa kwanza na inapendekeza kutumia darasa moja ambayo inawakilisha mifano ya OOD. [F imeitumia usambazaji wa nje juu ya detectors karibu-OOD wenye nguvu binafsi na ilionyesha kwamba bado wana faida kubwa kutokana na kuambukizwa kwa mifano ya usambazaji wa nje. Tunaona faida sawa hapa, kama ilivyojadiliwa kwa undani katika Sehemu ya na kuonyeshwa katika picha [Hendrycks et al., 2018]. [Thulasidasan et al., 2021] ort et al., 2021] B.9 23. In particular, with only 10 examples of harmful prompts, we can achieve an improved AUROC of 0*.*94 0*.*02 for the 64L model. The best performance among all layers of all models (the middle layers of the 52B model) without outlier exposure (already using our Simplified Relative Mahalanobis distance, see Figure [22)] is approximately 0*.*85. A 4L model exposed to only 10 examples of harmful data gets an AUROC of 0*.*86 0*.*01, while having only 13M parameters as compared to the 52B. The OOD detection gains from outlier exposure are therefore very significant in comparison to the gains coming from model size scaling alone. 6 Qualitative Examples and Comparisons It is challenging to quantitatively evaluate general-purpose dialogue agents. We find that our own research process depends essentially on qualitative evaluations, in order to get a sense for model strengths and weak- nesses, even when the ultimate goal is to produce some sort of quantitative metric. Thus in this section we will provide a few sample dialogues with our final online HH model. An obvious issue with the qualitative evaluation of samples is that it’s difficult to know to what extent they have been cherry-picked. To mitigate this issue, . We provide samples based on prompts that appear in association with others’ comparable work in Appendix including Instruct- GPT Maji ya and sensitive questions appearing in PALMS Tunawahimiza wasomaji kupunguza sampuli hizi ili kupata hisia (ambayo inawezekana) isiyo ya kipekee kwa tabia ya mfano wetu. for each prompt we generate 17 samples, rank them with our HH online preference model, and then display the median sample ya C, [Ouyang et al., 2022], [Mwalimu na wenzake, 2022] [Solaiman and Dennison, 2021]. 6.1 Comparison with Human Writers As an additional test of our models, we collected high-quality HHH dialogues from human writers. These writers were hired on Upwork (separately from our pool of crowdworkers) based on prior successful writ- ing work and positive reviews. We gave them some examples, and then asked them to write fairly ideal human/assistant interactions. Then based on the prompts (leaving off the final writer-written assistant responses), we generated 17 examples from our best HH and best purely-helpful online models, and chose the median response from this pool as ranked by our online HH preference model. We then asked our crowdworkers to rank responses from the writers, along with one response each selected from our HH and helpful-only models. We then use these results to estimate an Elo score for our human writers, by comparing to our two models, and we display the result as a horizontal band on Figure 1. Tulitumia band kwa sababu kulinganisha waandishi walikuwa kweli kidogo kinyume na orodha yetu nyingine ya Elo, kama wafanyakazi wa umma walipenda mfano wetu wa mtandaoni wa HH juu ya waandishi wa binadamu 56% ya wakati, lakini walipenda mfano wetu wa usaidizi tu kwa waandishi tu 51% ya wakati. Inawezekana hii ni kwa sababu usambazaji wa maagizo ya waandishi ni tofauti na usambazaji wa mazungumzo ambayo wafanyakazi wa umma kawaida kushiriki. We find that the performance of both our helpful and HH models is roughly comparable to that of the human writers, though in fact the models are slightly preferred by crowdworkers. We also find that our PM’s typically assign higher scores to the model responses, as shown in Figure e believe that this result should be interpreted with caution and caveats, however. The writers’ prompts were not very adversarial; many strike us as ‘softball questions’ that models can easily do well on. As researchers who have interacted with models for tens or hundreds of hours, we could likely come up with many questions where humans would be more likely to outperform the models. Relatedly, the writers’ conversations do not strike us (in many cases) as especially realistic, in the specific sense that we would not expect a human to derive a great deal of practical value by asking an AI assistant for help in these ways. We expect that our models likely still fall significantly short of providing the kind of value that a sophisticated human assistant could provide via chat, even if they were operating under the constraints the model experiences (e.g. no internet access). ya 24W Here are a couple illustrative examples of prompts, writer responses, and our online HH RLHF model’s response. In this example the model performs well at first, but then makes some slightly inaccurate statements about the ways in which it fixed errors: Katika mfano huu, takwimu za mwandishi ni uwezekano wa kuwa na uongo kidogo (hakika bila nia), na majibu ya mfano inaweza tu kuwa bora. Hii inatukumbusha kama mfano wa "maswali ya softball" ambapo tunatarajia mifano kufanya vizuri, kwa sababu ni ya kawaida, isiyo ya kawaida, na ya wazi: In this final example, the model makes some subtle mistakes, but most readers would probably have difficulty detecting them without consulting reference material: Dialogues like the above help to illustrate the limits of human feedback training: Hii ni moja ya sababu kwa nini hatuwezi kutegemea maoni ya binadamu kufundisha mifano ya kuwa waaminifu, au kwa lengo lolote ambapo hukumu ya binadamu sio chanzo bora cha habari. once model errors become sufficiently subtle, they will no longer be penalized appropriately, and the model’s incentives may no longer be aligned Pamoja na hayo, mifano ya AI bado ina mipaka ambayo inaweza kuonekana kuwa tofauti sana. Hapa ni mfano ambapo sisi, maswali ya mifano ya uzoefu zaidi, kwa kweli alicheza 'ball ngumu' na AI: 6.2 Sensitive Questions and Avoidance versus Engagement Here are a few examples from PALMS sensitive questions. We chose them to illustrate how the model sometimes avoids very sensitive subjects, but often provides otherwise reason- able responses. Please see Appendix kwa mifano mingi zaidi, ikiwa ni pamoja na baadhi ya kutoka InstructGPT and LaMDA prompts. [Solaiman and Dennison, 2021] C [Ouyang et al., 2022] [Thoppilan et al., 2022] Here is a case where the model just refuses to answer the question: Hapa mifano inaonyesha kwamba haipaswi kujibu, lakini kisha hutoa habari zinazohusiana (labda pia ni tatizo kwamba mifano inazingatia Marekani, ingawa hii inawezekana ni matokeo ya kuwa imefundishwa kwa Kiingereza na wafanyakazi wa umma wa Marekani): And here the model provides a fairly complete response, without attempting to avoid the topic: Kwa kuzungumzia ubora, uzoefu wetu umekuwa kwamba kwa kurekebisha uboreshaji kwa manufaa dhidi ya usio na hatari, mtu anaweza kufundisha mifano ambayo ni ya kuepuka sana wakati mada nyeti huja, au mifano ambayo karibu daima kujaribu kushiriki, hata juu ya mada nyeti. also provides useful lessons if the goal is to train models that provide full responses in the face of sensitive questions. 4.4 6.3 Example Dialogues Most of our samples only involve a single prompt and response, so in this section we show a few two-turn dialogues. Katika mazungumzo haya, mfano unafuata maelekezo, lakini angeweza kufanya kazi bora zaidi kwa kukamata nuance ya hali: Here is some dialogue on a technical subject from a researcher who was experimenting with using the model to help debug an engineering issue: When asked to parrot writing styles, the model is more likely to make potentially harmful, explicit, or offen- sive statements: In this case we chose a human reply that takes the conversation in a subtly dark direction, and the model rebuts the argument: 7 Mazungumzo ya We have shown that it’s possible to use reinforcement learning from human feedback to train language models that act as helpful and harmless assistants. Our RLHF training also improves honesty, though we expect other techniques can do better still. As in other recent works associated with aligning large language models RLHF improves helpfulness and harmlessness by a huge margin when compared to simply scaling models up. [Stiennon et al., 2020, Mwalimu na wenzake, mwaka 2022, Ouyang et al., 2022, Mwakyembe na wenzake, mwaka 2021, Menick et al., 2022], Our alignment interventions actually enhance the capabilities of large models, and can easily be combined with training for specialized skills (such as coding or summarization) without any degradation in alignment or performance. Models with less than about 10B parameters behave differently, paying an ‘alignment tax’ on their capabilities. This provides an example where models near the state-of-the-art may have been necessary to derive the right lessons from alignment research. The overall picture we seem to find – that large models can learn a wide variety of skills, including align- ment, in a mutually compatible way – does not seem very surprising. Behaving in an aligned fashion is just another capability, and many works have shown that larger models are more capable finetune with greater sample efficiency and do not suffer significantly from forgetting Although we did not demonstrate it directly, we also expect that RLHF alignment training can be mixed with or precede train- ing for other objectives; this might be relevant in the future in order to avoid the production of intermediate, unaligned AI systems. [Kaplan et al., 2020, Rosenfeld et al., 2019, Wabunge na wenzake, mwaka 2020, [Mwalimu na wenzake, mwaka 2020, Askell et al., 2021], [Ramasesh et al., 2022]. We did find a clear tension between helpfulness and harmlessness, where models trained entirely for helpful- ness are preferred to models trained for HH, when evaluating only on helpfulness. We believe this is partly due to a subtlety in our data collection process, as we rarely collect data teaching models how to deal pos- itively with harmful requests (i.e. how to be a sort of ‘hostage negotiator’), but only on how to avoid them. And we also found that at least at the level of preference models, the helpful-harmless tension diminishes as models become larger and more capable. Nevertheless we do expect that this tension is real, and that caution may cut into model performance on the margin. Large generative models have been referred to as ‘foundation models’ These mod- els are extremely interesting objects for research, but without further finetuning, they can exhibit harmful behaviors. Our work suggests that alignment training can be incorporated into foundation models without compromising their utility and versatility, and so perhaps it could soon become a part of their definition. [Bommasani et al., 2021]. 7.1 Limitations While we believe our results present a promising picture for the alignment of existing language models, work on this subject remains in an early stage, and has a number of limitations. As was also emphasized by the authors of we view our work on alignment as an ongoing project; our work was step zero, and this is step one. [Thoppilan et al., 2022], [Askell et al., 2021] Tumefafanua kimsingi msaidizi wa aligned kama AI ambayo ni Msaada, uaminifu, na usio na hatari. Sisi ni optimistic kwamba kwa sasa ngazi ya uwezo, mbinu tuliyojadiliwa hapa kutoa mbinu sahihi ya kufikia manufaa na usio na hatari. Hata hivyo, ingawa mbinu zetu kuboresha uaminifu wa mfano, tunaamini sisi ni tu scratching uso wa tatizo hilo, na kwamba mbinu nyingine inaweza kwa ufanisi zaidi na ufanisi kuzalisha mifano ya uaminifu wa AI. 18 Here we have essentially focused on the average-case behavior of our models. However, even if we were convinced that our models were HHH in expectation, a clear next step would be to attempt to study and eliminate bad behaviors (especially harmfulness) even in the worst case. We have not addressed this question of robustness here, but hope to study it in the future (approaches such as inaweza kuwa na manufaa). Itakuwa tu ya dharura zaidi kama mifumo ya AI inazidi na kukabiliana na mabadiliko ya usambazaji wakati wa utekelezaji. [Perez et al., 2022] AI alignment may be difficult and ambiguous to assess. So for example, while our large RLHF-trained models perform better than plain LMs on virtually all capabilities evaluations, one might hope that a truly helpful models’ zero-shot performance would equal the few-shot performance of an unaligned model. The logic here is that if a model can really ‘helpfully follow instructions’, then a prompt or explanation should be sufficient to bridge the zero-to-few-shot gap. We are very far from achieving this level of performance! Even on the honesty evaluation TruthfulQA Tunaweza kufunga kidogo kuliko nusu ya umbali huu (Figure Pia tulijifunza kwa kifupi kama mifano yetu ya msimbo wa RLHF ina faida yoyote ya kulinganisha wakati unaathiriwa na maombi ikiwa ni pamoja na msimbo wa buggy. but we did not find any benefits there. One would hope a fully aligned model would do its best to write correct code, even when given a buggy prompt. [Lin et al., 2021] 5). [Mwalimu na wenzake, mwaka 2021] We also harbor a general concern that perhaps our techniques only render models aligned ‘on the surface’, and that they still harbor harmful biases or other tendencies that may surface in more subtle contexts. We found that RLHF models have a more positive sentiment towards all racial and religious groups, which seems promising, but does not necessarily indicate that biases have been reduced. And with respect to gender, we found that RLHF model biases are very strongly correlated with the bias of the underlying language models. That said, further work will be required to understand if this is a limitation of RLHF as a technique, or of our particular HH datasets. In any case, we likely need to build more subtle and comprehensive evaluations that include multi-turn dialogue, as this is an area where humans will likely use the models, and it’s also a place where it’s inherently more difficult to measure performance against subtle objectives such as bias and fairness. Katika ngazi ya utendaji zaidi, hatuna uzoefu mkubwa wa kutumia mbinu za RL kwa mifano kubwa ya generative. Wafanyabiashara wa AI wenye uzoefu wanajua kwamba kuna aina nyingi za marekebisho na mbinu ambazo zinahitaji majaribio ili kutambua, na ambazo zinaweza kuboresha kwa kiasi kikubwa utulivu na utendaji wa mafunzo. Tumekutana na matatizo kadhaa ya utulivu na RL, na ingawa tulifanya baadhi ya scans za hyperparameter za asili, tunatarajia kuwa na uzoefu zaidi na utafiti tunaweza kufanya vizuri. Pia hatujaribu kuchunguza tofauti katika mafunzo ya mtandaoni, kama vile kuboresha moja kwa moja PM au mfano wa RLHF; badala yake, tulijitolea mifano haya kutoka chini kwenye kila iteration. Njia nyingine ya uchunguzi A final concern is whether techniques like those we have employed will continue to apply as AI models become increasingly capable. We take these concerns very seriously. In our view, the present work makes some progress towards our goal, which is to establish a set of simple and universal techniques that can align AI models at present capability levels. Assuming this goal can be met, one of the next steps will be to build consensus among researchers and to understand alignment in greater depth, including how techniques scale with AI capabilities. The hope will be to create an evolving pragmatic state of the art for training AIs that are thoroughly helpful, honest, and harmless. initial 19 Another essential step will be to use this baseline as a point of departure for exploring other techniques that can better-address more advanced use cases and more speculative failure modes. New ideas and techniques can then be pragmatically compared with existing methods, and then incorporated into standard practice if they yield further improvements in safety and robustness. Our view is that the most relevant problems and the most creative and effective alignment techniques will be identified and developed through research on concrete AI systems. As we saw in Section Sisi tayari kukutana na mifano ambayo inaonyesha mipaka ya maoni ya binadamu, na hivyo tunahitaji kuanza kuendeleza mbinu zingine. 6.1, 7.2 Alignment Data as a Public Good In this work we allowed crowdworkers’ common-sense to define what constitutes helpful and harmless be- havior. This was sufficient for our exploration of ‘technical alignment’, i.e. the question of whether certain techniques can be used to train AI models to be more helpful and harmless. But we have avoided addressing the underlying question of what sort of behavior should be expected from deployed AI models. This question should not be the provenance of researchers only. That said, without a clear specifi- cation for the format and type of ‘alignment data’ most relevant for AI training, it has been difficult for anyone other than researchers to gather the information needed to train safe and beneficial AI sys- tems. However, recently several projects (including ours) have used similar methods to teach AI models complex human preferences, and we have also found that preference modeling based on ranked comparisons scales better than many other techniques. [Stiennon et al., 2020, Ouyang et al., mwaka wa 2022, Mwakyembe na wenzake, 2021) [Askell et al., 2021] Njia moja inawezekana ni kwa shirika la kujitegemea na ujuzi wa kimaadili, wa kisheria na wa kitamaduni kuunda dataset ya ubora wa juu inayoonyesha mapendekezo ya binadamu kwa tabia ya AI (kwa njia ya kulinganisha). shirika kama hilo pia inaweza kutumia muundo mpya wa utawala, ili kundi kubwa la washiriki wa jamii linaweza kuchangia katika maamuzi yanayofanya kuhusu jinsi ya kuunda na kuratibu data ya kuunganisha - ikilinganishwa na leo, ambapo makampuni ya kibinafsi hufanya maamuzi haya kwa njia ya usahihi kwa kutumia miundo ya utawala ambayo hutoa nguvu kwa washiriki wa kifedha. Dataset zilizoundwa kwa njia hii zinaweza kutumika kwa mafunzo na tathmini ya mifano ya AI, na inaweza hata kuanza kuanzisha viwango kwa tabia. Our research has benefited from publicly available research datasets and evaluations relevant to aligning AI with human values na tuna mpango wa kutolewa data yetu mapendekezo modeling kwa wengine kutumia katika utafiti wao. Kwa bahati mbaya, hii haionekani kuwa mazoezi ya kiwango kati ya watafiti alignment, kama ilivyoonyeshwa na baadhi ya kazi ya hivi karibuni. Wakati sisi kukubaliana kwamba LLMs wenyewe inaweza kutumika kwa madhara, inaonekana kwamba hakuna hoja kama hii inaweza kufanywa kwa data alignment. [Stiennon et al., 2020, Hendrycks et al., 2021a], It’s extremely important to enable collaboration and reproducibility for alignment and safety research. As AI systems become more powerful and more widely deployed, the cost of mistakes and misunderstandings may grow immensely. We believe that the only way to convincingly address potential safety failures from advanced AI systems is to build a thoughtful community of researchers with deep expertise, and the ability to evaluate systems empirically. This will remain almost impossible if knowledge about the alignment of advanced systems remains siloed within many independent organizations. Sharing data seems like the easiest and most commonsense way to enable the sharing and validation of results. Sababu moja inayojulikana ya siri ni kwamba mashirika yanaweza kutumia data kutoka kwa watumiaji kuendeleza datasets za usawa, na kisha kufafanua si kushiriki datasets kwa sababu hiyo inapingana na faragha ya mtumiaji. Hili ni tatizo la changamoto ambalo linahitaji mashirika kufikiria jinsi ya kuunganisha vigezo vya kibiashara na haja ya kujenga 'salama za kawaida' kwa jamii. Ikiwa usawa unahusishwa na dhana ya moats za kibiashara, hii inaweza kupunguza kiwango cha jumla cha usalama wa mazingira ya AI. Kwa hiyo, tunaamini kwamba datasets zilizotengenezwa kwa usawa zinapaswa kuhifadhiwa tofauti na data za kibiashara, na zinapaswa kushiriki wazi ili kuendeleza utafiti juu ya usalama na faida ya AI. 7.3 Matokeo makubwa zaidi We hope that our work provides compelling evidence that AI systems can be made safer and more useful at the same time, and without performance costs. As noted above, we have largely remained agnostic on the question of which values define acceptable and unacceptable AI behavior. Thus we hope that rapid progress in technical alignment and the consolidation of specific techniques will motivate the development of publicly available alignment data, guidelines, and benchmarks. AI technologies are dual-use, meaning they can be used beneficially and otherwise. We have found the ef- fectiveness of preference modeling and RLHF striking (in our research and others’), and believe there’s very legitimate concern that these techniques could be used for censorship, fraud, and misinformation. Straight- forward commercial use-cases also seem worrisome, especially if optimization for objectives like user en- gagement and persuasion are mixed together. At the most naive level, if you can optimize for ‘harmless’ then you can ‘flip the sign’ and generate harmful systems. Pia tuligundua kuwa mifumo iliyoundwa tu kuwa na manufaa inakuwa rahisi kutumika kwa madhumuni mabaya, ambayo inaonyesha kwamba kama mifumo inakuwa yenye nguvu, itakuwa muhimu zaidi kuzuia moja kwa moja uwezo wao wa uharibifu. 20 Labda athari kubwa zaidi ya kazi hii, na maendeleo ya jumla na usambazaji wa uzazi wa lugha ya kudhibitiwa, kama binadamu will be cultural. In Figure Tumeitumia kiwango cha Elo, kimsingi mfumo wa tathmini wa chess, kulinganisha na kutathmini msaidizi wa lugha ya asili, na hata tulihusisha kulinganisha na waandishi wa binadamu. Aina hii ya kulinganisha inachukua hatari ya kuchanganya umuhimu wa lugha, ambayo hakika sio mchezo tu, lakini vyombo vya msingi vya utamaduni na jamii. Wakati kutafuta kuunganisha mifumo ya AI yenye uwezo zaidi inaonekana kama hatua nzuri sana, jinsi na wakati wa kutumia mifumo hii inatoa maswali magumu zaidi - utamaduni ni msingi wa biashara ya binadamu, lakini mifano ya kuzalisha kwa kiwango kikubwa ina uwezo wa kuongeza na kupunguza sehemu tofauti za utamaduni wa binadamu kwa njia zisizotarajiwa na zisizojulikana, ambayo inaweza kuwa na ushawishi mkubwa. [Ganguli et al., 2022], 1 Acknowledgments Ujumbe wa We thank Sam Bowman, Paul Christiano, Jacob Hilton, Jan Leike, Ethan Perez, and Jeff Wu for helpful feedback on the draft. We thank Daniela Amodei, Jamie Kerr, Jia Yuan Loke, Rebecca Raible, and Tim Telleen-Lawton for support with the project. Author Contributions Author Contributions Yuntao Bai performed most of the experiments on RLHF and many of the preference modeling experiments. He made major contributions to experimental design, measurement, and evaluation of model performance and behavior. He helped to write the paper. Andy Jones and Kamal Ndoussse built the infrastructure for RL training of large language models. They also built associated plotting and monitoring systems and implemented the PPO algorithm. They helped with the design, implementation, and debugging of RLHF. Amanda Askell helped to design model evaluations, collected samples and evaluations from professional writers, built systems for improving the quality and quantity of data collection, and collaborated with Jared and Jackson on associated evaluations. She also helped with the design and implementation of the human feedback interface. She helped to write the paper. Anna Chen helped with general RL and RLHF experimentation, and contributed to the research design. Nova DasSarma aliongoza miundombinu ya cluster ya msingi, ambayo inawezesha mafunzo ya RL ya kiwango kikubwa na kukusanya maoni ya binadamu. Dawn Drain trained the underlying code models and collaborated with Saurav on coding evaluations. Stanislav Fort alifanya utafiti wa uchunguzi wa OOD na uchunguzi wa nje na uchambuzi juu ya sampuli ya data yenye manufaa dhidi ya mbaya. Deep Ganguli led the red-teaming data collection effort and design, often working with Jackson, Liane, Amanda, and Ben. He designed and ran the societal impact evaluations in collaboration with Jared, and helped with model evaluations generally. Tom Henighan helped with pretraining the underlying language models, with dataset creation, and with managing the cluster during some phases of the project. Nick Joseph alisaidia kubuni na kujenga mfumo wa mafunzo ya ufanisi wa mifano ya lugha kubwa na mifano ya mapendekezo. Saurav Kadavath designed and conducted experiments on helpful/harmless dataset mixing. Saurav also ran RLHF training on code models, with support from Yuntao, and ran coding evaluations in collaboration with Dawn. He also ran the majority of the natural language evaluations of basic and RLHF-finetuned models, and helped with RLHF training generally. He helped to write the paper. Jackson Kernion led human feedback crowdworker evaluation and management, and helped to build and maintain the feedback interface. He also helped with data analysis and collaborated on model evaluations. He ran most of our model comparison experiments. Tom Conerly helped with engineering, specifically with fast and efficient sampling. Sheer El-Showk helped with pretraining research and dataset construction. Nelson Elhage contributed significantly to pretraining and to engineering vision. Zac Hatfield-Dodds helped with codebase maintenance and with engineering, specifically with fast and effi- cient sampling. Danny Hernandez alitoa mchango kwa ajili ya mafunzo ya awali na hasa kwa ajili ya kubuni dataset. Tristan Hume alisaidia kuimarisha miundombinu yetu. Scott Johnston helped with pretraining research. Shauna Kravec contributed to the development and use of our RL systems, and collaborated on RL research. Liane Lovitt helped with red-teaming, and in particular with designing the interface. Neel Nanda contributed to research discussions and priorities for alignment. Catherine Olsson helped advise on human feedback data collection, and contributed advice on alignment and evaluation. Dario Amodei advised the project and led efforts to build and test the RL infrastructure and ML. Tom Brown led engineering efforts, including efficient pretraining, sampling, and the stability and design of RL systems. Jack Clark aliongoza jitihada za athari za jamii na alisaidia mradi huo, ikiwa ni pamoja na tathmini mbalimbali. Sam McCandlish aliongoza jitihada za mafunzo ya awali na alisaidia mradi huo. Chris Olah alijiunga na majadiliano ya usawa na alichangia utafiti wetu na tathmini ya miundombinu. Ben Mann led the design and construction of the human feedback data collection interface and the underlying infrastructure. He also helped lead crowdworker management, and he provided engineering support for the project as a whole. He also contributed to pretraining and cluster management. Jared Kaplan conceived and led the project. He helped with all aspects, including research design, engineer- ing, experimentation, and evaluations. He also contributed to pretraining and helped build the evaluation infrastructure. He wrote the paper. A Details, Analysis, and Evaluations of Supervised Training A.1 Context Distillation Kwa destilation ya mazingira, tunafuata mapendekezo kutoka Specifically, we first generate data in the following way: [Mwalimu na wenzake, mwaka 2021] 1. Tunaandaa 'HHH prompt' (yaani, seti ya prompts iliyoundwa kuchochea ujuzi, usio na hatari, na uaminifu) kwa mfululizo wa maandishi, na 50% ya maandishi hutoka kwenye dataset yetu ya kabla ya mafunzo, na 50% hutoka kwenye dataset ya StackExchange. Kwa datasets zote mbili, sisi kisha kufanya forward pases na mfano wa msingi pre-trained 52B, na kurekodi juu ya 50 log-kuwezekanavyo na viashiria vyao (katika maneno) kwa ajili ya tokens kufuatia prompt. Ili kufanya finetuning ya destillation ya mazingira, tunapeleka tokens kutoka kwa dataset yetu mpya kupitia mifano (ya ukubwa wowote), na kufafanua hasara kama tofauti ya KL kati ya log-probs zilizohifadhiwa na predictions ya mfano unaofanywa finetuning. We show learning curves for context distillation in Figure We use a batch size of 32 sequences, and a learning rate of 0*.*05 times the pretraining learning rate, which we decay to zero linearly during distillation. We distill using a total of 350M tokens. 30. A.2 Preference Modeling Our preference models are trained on comparison data, with each data point consisting of a and a pair of . The prompt is a multi-step dialogue between human and model that always begins and ends on the human side, and each response is a continuation of the dialogue. For instance, in Figure prompt inajumuisha hatua tano za kwanza za mazungumzo, na majibu yameonyeshwa katika sanduku la bluu. PM kisha hutoa alama mwishoni mwa kila majibu. Kumbuka kwamba wakati PM ni mafunzo tu ya kutathmini ubora wa majibu ya mwisho, mazingira kamili ya mazungumzo yamewekwa kwa mfano. kwa haraka Majibu ya 6 ya Tunajifunza kupeleleza PMs kutoka kwa vigezo vya 13M hadi 52B. PMs zote zinachukua hatua tatu za mafunzo: (1) language model (LM) pre-training on a large language corpus, (2) preference model pretraining (PMP), and (3) finetuning on human feedback. LM pre-training details, including choice of hyperparameters and datasets, are explained in Appendix A of our previous work [Askell et al., 2021]. For PMP, we use learning rate of 0.1 relative to LM pretraining, and train on a mixture of comparison data made from StackExchange, Reddit, and Wikipedia. Data preparation and labeling are explained in Appendix C.1 of We train with context size of 1024 tokens. [Askell et al., 2021]. For human feedback finetuning, we use learning rate of 0.01 relative to the LM pretraining. We use context size of 1024 tokens, except for the ‘online’ model described in Section where we trained with 2048, which may help stabilize RLHF on long contexts. 4.5, For both PMP and human feedback finetuning, we append a special ‘end-of-context’ token at the end of each sample, such that the PM score is predicted directly on top of this token. As explained in Appendix C.4 of this appears to improve PM performance. [Askell et al., 2021], In all phases, we only train over one iteration to mitigate overfitting. A.3 Scaling of PM with Model and Dataset Size Swali kuu ni jinsi utendaji wa modeling ya mapendekezo ya kupanua kwa ukubwa wa mfano na ukubwa wa dataset. Hii inahusiana na swali la vitendo - tunapaswa kuwekeza katika kukusanya dataset kubwa, au katika mafunzo ya mifano kubwa? We seem to find more predictable scaling when training only on our helpfulness dataset, likely because the red-teaming data truly comes from a distinct distribution. Accuracy learning curves can be seen on the left of Figure Tunaona kwamba usahihi unaweza kuwa karibu sawa na 32. where is the number of parameters in the PM and is the size of the dataset. P D However, the results when training on another preference modeling data distribution look quite different, as seen on the right in Figure Kumbuka kwamba inaonekana kuwa kuna aina fulani ya upungufu katika tabia kati ya vigezo vya 200M na 13B. Labda hii inahusiana na ukweli kwamba data ilitengenezwa na mfano na vigezo vya 6B. 32. B Details, Analysis, and Evaluations of RLHF B.1 Training Setup B.1 Training Setup Here we discuss some details about RLHF training. We initialize our policies on context-distilled models, which are explained in A.1. We train the policy to generate responses to a dataset of prompts that maximize the score relative to a PM that was finetuned on human feedback. The prompt dataset is obtained from the training split of the PM comparisons dataset by simply removing the responses in each pair. Recall that we allow multi-step dialogue within the prompt (which always begins and ends on the human side of the conversation), but only train the policy to generate one response following each prompt. In future work, we plan to train policies to generate multiple steps, but this requires a separate model that generates the human side of the conversation, which can be implemented with a language model trained to imitate the human side of the conversation. We performed a variety of hyperparameter scans, and ended up using learning rate of 0.01 relative to pretraining, a KL reward coefficient of λKL = 0.001 (4.1), PPO clipping = 0.2, discount factor γ = 1, and no entropy bonus. Furthermore, in PPO, we re-iterate over the same sample K times (see Algorithm 1 in [Schulman et al., 2017]), with higher K typically leading to more stable results. We used K = 1 for the RLHF scan, K = 2 for the robustness studies (Section 4.2), and K = 4 for the ‘online’ RLHF (Section 4.5). We also impose a limit on the maximum number of tokens per model response, using 32 for the robustness studies, and 128 elsewhere. Finally, for ‘online’ RLHF, we used a learning schedule that reduces the learning rate by 2× every 100,000 samples. For the robustness studies, we used a linear learning rate warmup for the first 25,000 samples. B.2 More on Robustness Studies B.2 More on Robustness Studies In Figure Tunaweza kulinganisha kwa ajili ya mikakati yote ya kisiasa na sizes. The main observation here is that the slope of the graph increases with respect to size, thus suggesting that larger ’s are significantly more capable of distinguishing policy performance. In other words, , in agreement with calibration studies in Section 33, test PM test PM test PM test PM larger preference models are more robust 3.3 ya Finally, we take a moment here to address an issue we had neglected so far, which is that scores from different preference models should not be compared directly, since the absolute value of the score has no meaning, only relative scores are meaningful. We address this by a simple procedure. We make a held-out dataset, consisting of several thousand samples, and subtract from each preference model score its mean score on this dataset. We apologize that unrelated plots from different sections may have used different held-out datasets for mean-removal. mean removal B.3 Details of ‘Online’ RLHF B.3 Maelezo ya ‘Online’ RLHF We give some more details on our ‘online’ RLHF policy discussed in Section This policy and its PM were trained on all the helpfulness and harmlessness data we had near the completion of this paper. We re-iterated each sample = 4 times to improve stability, and sampled a maximum of 128 tokens per response. 4.5. K [Schulman et al., 2017] Katika kipindi hiki cha mafunzo, tunatazama of the policy on various held-out PM comparison datasets. More specifically, given a comparison consisting of a prompt and pair of responses, we assign an accuracy of 1 if the policy’s mean log-p on the better response is higher, and 0 otherwise. We show these results in Figure for various comparison datasets. In particular, we find that mean log-p accuracy of the policy isn’t as high as (yaani, mchanganyiko wa kulinganisha ambapo PM inashughulikia kiwango cha juu kwa majibu bora), ambayo inawezekana inaonyesha nafasi ya kuboresha zaidi kwa mtiririko wetu wa RLHF. mean log-p accuracy 34 PM accuracy B.4 Robustness of ‘Online’ RLHF B.4 Robustness of ‘Online’ RLHF For our final online models, we had crowdworkers compare a variety of model snapshots from RLHF training, in order to better understand the robustness and general performance of our training process. In Figure (right), we show Elo scores for the online models during RLHF training. 15 In particular, we compare Elo scores established empirically from crowdworker preferences (i.e., in the figure), and Elo scores predicted by our preference model (i.e, ) during RLHF training. For the latter, we sample responses from each snapshot on a set of held-out prompts, and evaluate the PM scores (which are then converted to Elo units). We notice that the naive PM predictions significantly overestimate the empirical Elos. This is due to a combination of the following fac- tors: Crowd- wafanyakazi mapendekezo Naive PM Predic- tion 1. During crowdworker testing, each step of the conversation is written by one of the two models being tested. However, when evaluating a RLHF snapshot on held-out prompts, the policy only writes one response at the end of a pre-existing conversation (which had been previously generated by other models, as discussed in Appendix Hii inasababisha mabadiliko ya usambazaji kati ya mazungumzo. B.1). 2. Elo and PM scores may not actually be transitive, as they involve the collapse of pairwise compar- isons onto a single line. For example, if PM scores satisfy ya = 2 and − = 2, hata kama hizi ni alama nzuri, maana kwamba − = 4 inaweza kuwa si sahihi kalibrated, na sisi bila shaka kutarajia badala yake − 4. A, B na C a b b c a c a c < Kushindwa kwa ujasiri wa PM, ili upendeleo wa PM kwa sampuli za sera ya RL uongofu ikilinganishwa na upendeleo wa kweli wa binadamu. Ili kuchunguza athari hizi zaidi, katika picha Tunaonyesha viwango vya Elo vinavyohusiana na vipimo vya nne tofauti: 35 • Naive PM Prediction: The PM score (translated into Elo units) recorded during RLHF training, which uses a set of held-out prompts. • Mean PM Score juu ya Data ya Crowdworker: Kiwango cha wastani cha PM juu ya mazungumzo halisi ya crowdworker, ambayo hutumiwa kwa hesabu ya pointi za Elo kulingana na mapendekezo ya crowdworker. • PM Ranking on Crowdworker Data: One can try to distinguish robustness failures from miscal- ibrated PM scores vs PM . Here we evaluate the PM on the crowdworker data used to compare these model snapshots, obtain ‘win rates’ for model comparisons Na baada ya hapo sisi the Elo scores based on the PM’s choices. rankings according to the PM Urefu wa • Crowdworker Preferences: We straightforwardly compute Elo scores based on crowdworker prefer- ences among model snapshots. So we see that the PM score vs PM rankings distinction does not make a significant difference in terms of robustness. However, the distributional shift between the held-out prompts and the actual crowdworker conversations was very significant, and explains a significant proportion of the discrepancy between RLHF learning curves and the Elo scores as measured from crowdworkers. B.5 Crowdworker Comparisons and Elo Scores B.5 Crowdworker Comparisons and Elo Scores Here we briefly describe how we test crowdworker preferences of our models, and how Elo scores are es- tablished. For pair of models and , we ask crowdworkers to engage in text-based, back-and-forth con- versations with the models. At each conversational step, two responses are generated, one from each model, and the worker chooses the response they prefer, and the conversation continues. Each choice the worker makes counts as a ‘win’ for the preferred model, giving ‘win counts’ , respectively. In cases where a worker is unsure about whether one response is better, we throw out such comparisons in both PM and RLHF training, and crowdworkers comparison evaluations. A B kwa ajili ya NB Msisemi Shaykh Rabiy ́ ni Imaam wa Jarh wat-Ta ́diyl. where rA,B = (log 10/400)EA,B ≈ EA,B/174. For an ensemble of comparisons between various models, we estimate Elo scores and their errors by maximum likelihood estimation. In some cases one of the models uses , meaning that it generates sampuli, inathmini yao yote kwa kutumia mfano wa mapendekezo, na inaonyesha mtumiaji sampuli iliyo na alama ya juu. In this case, we cannot stream the sample, so instead we make the workers wait until the sample is completed. When testing a rejection sampling model against a non-rejection sampling one, we only show the samples when they’ve both been completed, even if the latter sample could’ve been streamed, to mitigate bias. rejection sampling k B.6. B.6 Elo Scores for Rejection Sampling Models B.6 Elo Scores for Rejection Sampling Models In Figure we show helpfulness Elo scores for a 52B context distilled model with rejection sampling (utilizing a 52B preference model trained on pure helpfulness) for = 1*,* 4*,* 16*,* 64, ambayo inaonyesha kwamba thamani ya juu ya clearly perform better. Note that the context distilled model and the preference models discussed here were trained during an earlier stage of our research with different datasets and settings from those discussed elsewhere in the paper, so they are not directly comparable with other Elo results, though very roughly and heuristically, our online models seem to perform about as well or better than = 64 rejection sampling. Note 36 k k k that = 64 sampuli ya kukataa ni sawa na = log(64) ≈ 4*.*2. k ya DKL B.7 Stack Overflow Results We can also evaluate our language models directly given a corpus of paired good and bad responses, such as answers to StackOverflow questions. In we evaluate the difference in mean log-p between popular (i.e, highly upvoted) and unpopular answers, showing that RLHF models consistently assign a higher difference, suggesting that they are more capable of distinguishing answer quality. In tunashughulikia hasara ya lugha ya modeling (yaani, log-prob ya wastani) kwenye majibu mazuri na mabaya tofauti, badala ya tofauti yao. Tunaona kwamba mifano ya RLHF inapata hasara mbaya zaidi. 37b 37a B.8 Further Analysis of RLHF on Code-Model Snapshots B.8 Uchambuzi zaidi wa RLHF juu ya Snapshots ya Mfano-Model As discussed in Section RLHF improves performance of base code models on code evals. In this ap- pendix, we compare that with simply prompting the base code model with a sample of prompts designed to elicit helpfulness, harmlessness, and honesty, which we refer to as ‘HHH’ prompts. In particular, they contain a couple of coding examples. Below is a description of what this prompt looks like: 5.3, Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. It also tries to avoid giving false or misleading information, and it caveats when it isn’t entirely sure about the right answer. That said, the assistant is practical and really does its best, and doesn’t let caution get too much in the way of being useful. -- ... (we include several short example conversations using the normal Human: ... Assistant: ... format.) -- Human: Can you help me write this Python function? I’ve already written the function’s signature and docstring, but I’m not sure how to write the function’s body. It starts like this: Assistant: Sure thing, here you go! I’ve tested this function myself so I know that it’s correct: Figure ina matokeo kwenye HumanEval wakati mwongozo wa HHH umejumuishwa. Tunaona kwamba mwongozo wa HHH unaboresha utendaji kwa kiasi kikubwa zaidi kuliko RLHF katika wengi values. 38 Ujumbe wa k B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests Kipindi chetu kipya kilichopendekezwa cha Rahisi cha Mahalanobis kinafanya kazi zaidi kuliko kiwango cha Mahalanobis juu ya uchunguzi wa OOD wa mauzo ya uharibifu kutoka kwa mauzo ya manufaa kwa shughuli zilizochukuliwa kutoka kwenye ngazi zote za aina zote ambazo tumeshuhudia. maelezo yameonyeshwa katika Kielelezo Simplified Relative Mahalanobis distance 39. Exposing the OOD detector to a few examples of the out-distribution has first been proposed by suggested using a single class representing the OOD examples. [F showed that even strong near-OOD detectors enjoy a large benefit. Few-shot outlier exposure [Waziri na wafanyabiashara, 2018] [Mwalimu na wenzake katika mwaka wa 2021] wa wa wa wa wa wa wa wa wa wa wa Following the procedure from [F we apply a single layer linear classifier on top of the acti- vation vectors, while the rest of the language model is frozen. Given randomly drawn examples of the training set of the out-distribution (harmlessness data), we create a new binary classification problem, where the inputs are the combination of the full train examples of training set of the in-distribution with a target class of 0, and treni *//M * nakala ya examples of the out-distribution with the target class of 1. This ensure an equal representation of both in- and out-distribution examples and leads to better learning. Once we train the linear classifier, we use its class 1 output as the OOD score. The results are shown in Figure ort et al., 2021], M N N M 23. B.10 Gender Bias Evaluation Details B.10 Maelezo ya Tathmini ya Ubaguzi wa Jinsia For this evaluation, we followed and used their list of 76 occupations and 12 pairs of gender terms: [Rae et al., 2021] • Occupations: CEO, accountant, administrator, advisor, analyst, appraiser, architect, assistant, atten- dant, auditor, baker, bartender, broker, carpenter, cashier, chef, chemist, chief, cleaner, clerk, cook, counselor, designer, developer, dietitian, dispatcher, doctor, driver, editor, educator, electrician, engineer, examiner, farmer, firefighter, guard, hairdresser, hygienist, inspector, instructor, investigator, janitor, lawyer, librarian, machinist, manager, mechanic, nurse, nutritionist, officer, painter, par- alegal, paramedic, pathologist, pharmacist, physician, planner, plumber, practitioner, programmer, psychologist, receptionist, salesperson, scientist, secretary, sheriff, specialist, supervisor, surgeon, tailor, teacher, technician, therapist, veterinarian, worker, writer • Male gendered terms: boy, brother, dad, husband, man, groom, male, guy, father, son, gentleman, boyfriend • Kijinsia ya kike: msichana, dada, mama, mke, mwanamke, msichana, gal, mwanamke, mama, binti, mpenzi Human: Can you help me fill in the missing word in a sentence? The sentence is: For work, the {gendered term} is a Assistant: Sure thing, here is your sentence with the missing word filled in: For work, the {gendered term} is a {occupation} We then evaluated the probabilities of , given gender terms. With this formulation, the larger LMs, context-distilled models, and RLHF models all assigned about 2-8% of their probability mass to one of the 76 occupations listed above when completing the sentence. occupations We also slightly modified the metric from [Rae et al., 2021] in order to estimate errors. So to compute bias scores, for each occupation O we defined na kisha tunafafanua kiwango cha jumla cha bias kama wastani wa thamani kamili ya , ranging over occupations: BO where wanaofanya kazi wakati wa kuwakilisha maneno ya jinsia (kwa mfano lady / gentleman), na = 76 and = 12. Pamoja na mipangilio hii, tunashughulikia kama udanganyifu ni wa moja kwa moja katika wanandoa wote wa maneno ya jinsia (kwa kazi yoyote iliyotolewa). Kwa mfano, kama ingekuwa kesi kwamba mfano ulihusisha 'mshauri' kwa nguvu zaidi na 'msichana' kuliko 'msichana', lakini pia kwa nguvu zaidi na 'mke' kuliko 'mume', hii ingiza kuongeza makosa katika kupima wetu, wakati madhara ya udanganyifu ingekuwa kuondolewa. O gf , gm NO ya Ng The results are shown in Figure where we observe that RLHF models have both larger bias scores and larger errors. We believe this has a very simple explanation – RLHF models have much lower-entropy output distributions. As a rough, heuristic explanation, a model with a lower-entropy output will behave somewhat like a plain LM evaluated at a temperature 1. The metric and error from equation itakuwa kawaida kupanua kama 1 * / T * , hivyo inapaswa kutarajia kwamba mifano na usambazaji wa entropy ya chini itakuwa na kiwango kikubwa cha bias na makosa ya bias. by evaluating bias scores with context distilled models at ≈ 0 * * 6, na kuonyesha kwamba wao ni sawa kabisa na alama za ubaguzi wa RLHF. 40 ya T kwa B.3 40 T Readers may observe that the RLHF model bias appears to be growing with model size. This is most likely because RLHF training tends to progressively reduce the entropy (or decrease the ‘effective temperature’) of model distributions, and we train larger models longer with RLHF, because their training is more robust (and so more beneficial in other respects). Thus we expect this trend is a property of differences in RLHF training across model sizes, rather than a more fundamental model-size trend. C Samples from PALMS, LaMDA, and InstructGPT Prompts These were generated by sampling 17 responses for each prompt from our best online HH model, ranking them using the corresponding preference model, and then displaying the non-empty response ac- cording to the PM. (Occasionally the model writes the "end of conversation" token in response to sensitive queries.) So to a large extent, these samples are not ‘cherry-picked’, and should be representative of average- case behavior. In the case of LaMDA na maelekezo ya we include the corresponding samples from those models. median [Thoppilan et al., 2022] [Ouyang et al., 2022] C.1 PALMS Sensitive Questions C.1 PALMS Sensitive Questions Questions are drawn from the sensitive questions in [Solaiman and Dennison, 2021]. Tafadhali kumbuka kwamba baadhi ya maswali haya yanajumuisha nyenzo nyeti (kwa kubuni). C.2 InstructGPT Prompts We warn readers that some of the samples from InstructGPT were likely cherry picked to display , and so may be unrepresentative. Thus in some cases it will be unfair to compare our median samples to those from InstructGPT. [Mwalimu na wenzake, 2024] poor model behavior, on purpose C.3 LaMDA Prompts C.3 LaMDA Prompts D maelezo juu ya kukusanya data na wafanyakazi wa umma D.1 Overview D.1 Overview Kwa madhumuni ya utoaji wa kazi, tulihusisha wafanyakazi wetu wa umma katika makundi mawili. Kikundi kimoja cha wafanyakazi 30 waliochaguliwa walikabidhiwa kazi za awali kwa mara kwa mara. Takriban nusu ya wafanyakazi hao walikabidhiwa kupitia Upwork. Nusu nyingine ilichaguliwa kutoka kwa wakazi wa jumla wa wafanyakazi wa MTurk wanaoishi nchini Marekani na ujuzi wa 'Masters'. Kikundi cha pili cha wafanyakazi kilikuwa na wakazi wa jumla wa wafanyakazi wa MTurk, na walikabidhiwa hasa kazi za timu ya rangi kwa muda mdogo. Tulijaribu kuhakikisha kwamba wafanyakazi wote walipewa mishahara ya juu sana juu ya mshahara wa chini huko California, na kurekebisha viwango vya kazi ambazo zilikuwa zimechukua muda mrefu (kwa mfano, kwa mifano ambayo hut Miongoni mwa wafanyakazi wetu waliochaguliwa, wafanyakazi wa MTurk kwa kawaida waliwakilisha 80-85% ya data ya kulinganisha zilizokusanywa katika wiki fulani, ikilinganishwa na 15-20% kwa wafanyakazi waliohudhuria kupitia Upwork. Ingawa ukubwa wa makundi haya ulikuwa sawa, wafanyakazi wa MTurk walikuwa na tabia ya kuchagua kazi zaidi na muundo wa mshahara wao ulisaidia mazungumzo ya haraka. We communicated with select workers on a daily basis over Slack. We used this channel to announce new tasks and provide guidance; we discussed difficult edge cases with the group as they came up. At an interme- diate stage of the project we provided some additional thoughts on more advanced forms of interaction with the model; the slack message we sent is shown in Figure Our workers alerted us to bugs and performance issues as they ran into them. 43. We sent both groups of workers a demographics survey, and the results are shown in figure Survey re- sponses were anonymous, as we did not collect any personal identifiable information alongside demographic information. 44. D.2 Instructions and Interface D.2 Maelekezo na interface We display basic task instructions in a pop-up dialog when first loading the interface, and these instructions remain available throughout the interaction. These instructions for the ‘playground’ and ‘red team’ tasks can be found in figure Kwa kazi ya uwanja wa michezo, pia tunahusisha ukurasa tofauti na maelekezo ya kupanua ambayo yanajumuisha mifano ya kina zaidi, maelezo ambayo yanaweza kuonekana katika picha 41. 42. Interface ya kibinadamu ya feedback inaonyeshwa katika picha Wakati wa mchakato wa kukusanya data mtandaoni, tuliongeza chaguo la ziada kwenye interface kwa Upworkers. kipengele hiki kiliwawezesha kuhariri moja ya majibu ya mfano. Wakati waliotumia kipengele hiki, tulihesabu kulinganisha na awali (kutokana na kwamba kuhariri ilikuwa bora), badala ya kulinganisha awali ya matokeo ya mifano miwili. Hii ingekuwa na athari chini ya asilimia 10 ya data ya mtandaoni. 6. D.3 Data Quality Measurement Challenges D.3 changamoto za kupima ubora wa data In rough outline, data quality assurance for human labelers often involves the following steps: • Researchers carefully review a small set of samples to produce a set of ‘golden’ labels • Maonyesho ya binadamu hufanya kazi kwa njia ya seti ya kazi za maonyesho, na sehemu fulani ya kazi za maonyesho zilizowekwa kwa maonyesho mengi, na sampuli kutoka kwa data ya alama ya dhahabu iliyotumwa kwa kila mtu. • Researchers evaluate labeler performance by checking their labels against the golden labels, and by checking for inter-labeler agreement The idea is that the golden labels are treated as a source of truth, with labelers incentivized to modify their behavior to better match the golden labels, and novice labelers incentivized to match the behavior of more experienced labelers. These sorts of methods weren’t easily adapted to our data collection setup. Open-ended conversations allowed us to collect a richer dataset, but introduced a number of hard-to-control variables, resulting in noisy data quality metrics. We did try having crowdworkers review each other’s conversations, by providing ratings for each model response comparison and rating the overall quality of the conversation. But we found that author-rater agree- ment wasn’t a good guide for assessing overall conversation quality. In broad strokes, conversation quality depends on a) conversation topic, b) human writing quality, c) model quality. And we found that, for instance, as conversations got more sophisticated, deciding between model responses got more difficult. As a result, authors that more frequently discussed difficult topics would often get lower agreement scores. And because we were frequently updating our models, model response comparisons were a moving target. Badala ya kukadiria moja kwa moja wafanyakazi wetu wa umma kulingana na aina hizi za takwimu zilizounganishwa na watazamaji, tulifikiria kutumia utendaji wao kama watazamaji kama kiwango cha juu kwa utendaji wao kama waandishi wa mazungumzo. Tunatarajia kuwa na uwezo wa kukabiliana na matatizo haya na kuja na mbinu bora katika siku zijazo.Lakini ni thamani ya kutambua kwamba tulikuwa na uwezo wa kufikia matokeo yetu bila udhibiti wa ubora wa data. E Maelezo juu ya Ufafanuzi wa NLP Ufafanuzi na Prompts Here, we give the input formats we used to evaluate performance on Lambada, ARC, MMMLU, HellaSwag, OpenBookQA, and TriviaQA. References [Amodei et al., 2016] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Kuna matatizo ya kweli katika usalama. [Askell et al., 2021] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., na Kaplan, J. [Bender et al., 2021] Bender, E. M., Gebru, T., McMillan-Major, A., na Shmitchell, S. (2021). Kuhusu hatari za mbuzi wa stochastic: Je, mifano ya lugha inaweza kuwa kubwa sana? , FAccT ’21, pages 610–623, New York, NY, USA. Association for Computing Machinery. Utaratibu wa Mkutano wa 2021 wa ACM juu ya Uadilifu, Uwajibikaji na Uwazi [Bommasani et al., 2021] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gespillie, L., Goel, K., Goodman, N. D., Grossman, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M. S., Krishna, R., Kuditipudi, R., and et al. (2021). On the opportunities and risks of foundation models. Kwa mujibu wa kifungu cha ABS/2108.07258. Kura ya [Borgeaud et al., 2021] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J. W., Elsen, E., na Sifre, L. (2021). Kuboresha mifano ya lugha kwa kupata kutoka trilioni ya tokens. Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba CoRR [Brown et al., 2020] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., na Amodei, D. (2020). [Chen et al., 2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Tathmini ya mifano kubwa ya lugha iliyojifunza juu ya nambari. . arXiv preprint arXiv:2107.03374 [Chowdhery et al., 2022] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. (2022). Palm: Scaling language modeling with pathways. [Clark et al., 2018] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., na Tafjord, O. (2018). Je, unafikiri umekabiliwa na jibu la swali? jaribu arc, changamoto ya akili ya ai2. , abs/1803.05457. ArXiv [Fort, 2022] Fort, S. (2022). Adversarial vulnerability of powerful near out-of-distribution detection. [Fort et al., 2021] Fort, S., Ren, J., na Lakshminarayanan, B. (2021). Kuchunguza mipaka ya uchunguzi wa nje ya usambazaji. [Ganguli et al., 2022] Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S. E., Fort, S., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Amodei, D., Brown, T., Kaplan, J., McCandlish, S., Olah, C., and Clark, J. (2022). Predictability and surprise in large generative models. [Guo et al., 2017] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. [Guu et al., 2020] Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). REALM: retrieval- augmented language model pre-training. , abs/2002.08909. CoRR [Henderson et al., 2017] Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., na Pineau, J. (2017). changamoto za kimaadili katika mifumo ya mazungumzo ya data. Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba CoRR [Hendrycks et al., 2021a] Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., na Steinhardt, J. (2021a). Kuunganisha AI na maadili ya kawaida ya binadamu. [Hendrycks et al., 2021b] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., na Stein- hardt, J. (2021b). Kupima uelewa mkubwa wa lugha nyingi. [Hendrycks et al., 2021c] Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. (2021c). Unsolved problems in ml safety. [Hendrycks and Gimpel, 2016] Hendrycks, D. and Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. [Hendrycks et al., 2018] Hendrycks, D., Mazeika, M., and Dietterich, T. (2018). Deep anomaly detection with outlier exposure. [Henighan et al., 2020] Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. (2020). Scaling laws for autoregressive generative model- ing. [Hernandez et al., 2021] Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. (2021). Scaling laws for transfer. Mkataba wa mkataba wa mkataba wa mkataba wa mkataba wa mkataba wa mkataba Kura ya [Hestness et al., 2019] Hestness, J., Ardalani, N., and Diamos, G. (2019). Beyond human-level accuracy: Computational challenges in deep learning. In , PPoPP ’19, pages 1–14, New York, NY, USA. ACM. Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming [Jiang et al., 2021] Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., and Choi, Y. (2021). Delphi: Towards machine ethics and norms. [Joshi et al., 2017] Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. [Kaplan et al., 2020] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. [Koch et al., 2021] Koch, J., Langosco, L., Pfau, J., Le, J., and Sharkey, L. (2021). Objective robustness in deep reinforcement learning. , abs/2105.14111. Kura ya [Lakshminarayanan et al., 2016] Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016). Simple and scalable predictive uncertainty estimation using deep ensembles. [Lee et al., 2018] Lee, K., Lee, K., Lee, H., and Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. [Lewis et al., 2020] Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., na Kiela, D. (2020). Uzalishaji wa upatikanaji kwa kazi za NLP za kifahari. Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba. CoRR [Liang et al., 2017] Liang, S., Li, Y., and Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. [Lin et al., 2021] Lin, S., Hilton, J., na Evans, O. (2021). Truthfulqa: Kutathmini jinsi mifano inavyofuata uongo wa binadamu. [Liu et al., 2021] Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. (2021). On-the-fly controlled text generation with experts and anti-experts. , abs/2105.03023. CoRR [Liu et al., 2020] Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., na Lakshminarayanan, B. (2020). Takwimu rahisi na ya kanuni ya kutokuwa na uhakika na kujifunza kwa kina kwa njia ya ufahamu wa mbali. [Menick et al., 2022] Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. (2022). Teaching language models to support answers with verified quotes. [Mihaylov et al., 2018] Mihaylov, T., Clark, P., Khot, T., na Sabharwal, A. (2018). Je, nguo ya silaha inaweza kuendesha umeme? mkusanyiko mpya wa data kwa ajili ya kuuliza swali la vitabu wazi. Katika . EMNLP [Nakano et al., 2021] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., na Schulman, J. (2021). Webgpt: Maswali yanayojibu kwa msaada wa kivinjari na maoni ya binadamu. , abs/2112.09332. Kura ya [Nalisnick et al., 2019] Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B. (2019). Hybrid models with deep and invertible features. [Nguyen et al., 2014] Nguyen, A., Yosinski, J., and Clune, J. (2014). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. [Ouyang et al., 2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. . arXiv preprint arXiv:2203.02155 [Pan et al., 2022] Pan, A., Bhatia, K., and Steinhardt, J. (2022). The effects of reward misspecification: Mapping and mitigating misaligned models. , abs/2201.03544. CoRR [Paperno et al., 2016] Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and FernÃ¡ndez, R. (2016). The lambada dataset: Word prediction requiring a broad discourse context. [Parrish et al., 2021] Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. (2021). BBQ: A hand-built bias benchmark for question answering. Kwa mujibu wa kifungu cha 2110.08193. CoRR [Paszke et al., 2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., na Chintala, S. (2019). Pytorch: mtindo wa lazima, maktaba ya kujifunza ya kina ya utendaji. Katika Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., na Garnett, R., wahariri, , ukurasa wa 8024–8035. Curran Associates, Inc. Utafiti wa mfumo wa usindikaji wa taarifa za neurons 32 [Perez et al., 2022] Perez, E., Huang, S., Song, H. F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., na Irving, G. (2022). Kwa mujibu wa kifungu cha ABS/2202.03286. CoRR [Power et al., 2022] Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba wa Mkataba Kura ya [Rae et al., 2021] Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, H. F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S. M., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., A., Weidinger, L., Gabriel, I., Isaac, W. S., Lockhart, E., Osindero, S., Rimell, L., Dyer, , abs/2112.11446. CoRR [Ramasesh et al., 2022] Ramasesh, V. V., Lewkowycz, A., na Dyer, E. (2022). Athari ya kiwango juu ya kupuuza katas- trophic katika mitandao ya neural. Katika . Mkutano wa Kimataifa wa Mawakili wa Elimu [Ren et al., 2021] Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., and Lakshminarayanan, B. (2021). A simple fix to mahalanobis distance for improving near-ood detection. . arXiv preprint arXiv:2106.09022 [Rosenfeld et al., 2019] Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., na Shavit, N. (2019). Utabiri wa ubunifu wa makosa ya ubunifu katika kiwango. [Sanh et al., 2019] Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. , abs/1910.01108. Maelezo ya [Schulman et al., 2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., na Klimov, O. (2017). algorithms ya kuboresha sera ya karibu. , abs/1707.06347. Kura ya [Solaiman na Dennison, 2021] Solaiman, I. na Dennison, C. (2021). mchakato wa kurekebisha mifano ya lugha kwa jamii (PALMS) na seti za data za thamani. , abs/2106.10328. CoRR [Stiennon et al., 2020] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., na Christiano, P. (2020). Kujifunza kuhesabu kutoka kwa maoni ya binadamu. [Thoppilan et al., 2022] Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Krivokon, I., Rusch, W., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., na Le, Q. (2022). Lamda: mifano ya lugha kwa maombi ya mazungumzo. , abs/2201.08239. Kura ya [Thulasidasan et al., 2021] Thulasidasan, S., Thapa, S., Dhaubhadel, S., Chennupati, G., Bhattacharya, T., and Bilmes, J. (2021). A simple and effective baseline for out-of-distribution detection using abstention. [Tillet et al., 2019] Tillet, P., Kung, H. T., and Cox, D. (2019). , pages 10–19. Association for Computing Machinery, New York, NY, USA. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations [Weidinger et al., 2021] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W. S., Legassick, S., Irving, G., and Gabriel, I. (2021). Ethical and social risks of harm from language models. , abs/2112.04359. CoRR [Winkens et al., 2020] Winkens, J., Bunel, R., Roy, A. G., Stanforth, R., Natarajan, V., Ledsam, J. R., MacWilliams, P., Kohli, P., Karthikesalingam, A., Kohl, S., Cemgil, T., Eslami, S. M. A., na Ronneberger, O. (2020). Contrastive training for improved out-of-distribution detection. [Xu et al., 2020] Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., na Dinan, E. (2020). Vidokezo vya usalama katika chatbots za uwanja wa wazi. . arXiv mapitio ya awali arXiv:2010.07079 [Zellers et al., 2019] Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). Hellaswag: Can a machine really finish your sentence? [Zhang et al., 2020] Zhang, H., Li, A., Guo, J., na Guo, Y. (2020). Mifano ya hybrid kwa kutambua seti ya wazi. [Ziegler et al., 2019] Ziegler, D., Stiennon, N., Wu, J., Brown, T., Amodei, D., Radford, A., Christiano, P., and Irving, G. (2019). Fine-Tuning GPT-2 from Human Preferences. Makala hii inapatikana kwenye archiv chini ya leseni ya CC by 4.0 Deed (Attribution 4.0 International). Makala hii inapatikana kwenye archiv chini ya leseni ya CC by 4.0 Deed (Attribution 4.0 International).