AI berguna dan tidak berbahaya: Latihan Alignment meningkatkan prestasi pada hampir semua penilaian NLP

Latihan pembantu yang berguna dan tidak berbahaya dengan pembelajaran penguatkuasaan daripada maklum balas manusia Penulis : Yuntao Bai oleh Andy Jones Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Hari Raya Drain Stanislav Kuasa Gangguan yang mendalam Tajuk : Henighan oleh Nicholas Joseph Saurav Kadavath Jackson Kernion Tajuk : Tom Conerly Sheer El-Showk Nelson Elhage Zac Hatfield-Dodds oleh Danny Hernandez oleh Tristan Hume Scott Johnston Shauna Kravec Rakan Lovitt Melayu Nanda Catherine Olsson Dario Amodei Tom Brown oleh Jack Clark Sam McCandlish Chris Olah Ben lelaki Jared Kaplan Penulis : Yuntao Bai oleh Andy Jones Kamal Ndousse Anjing Askell oleh Anna Chen Dasar Baru Hari Raya Drain Stanislav Kuasa Gangguan yang mendalam Tajuk : Henighan oleh Nicholas Joseph Saurav Kadavath Kernion daripada Jackson Tajuk : Tom Conerly Sheer El-Showk Nelson Menghadapi Beliau ialah Zac Hatfield-Dodds oleh Danny Hernandez oleh Tristan Hume oleh Scott Johnston Shauna Kravec Rakan Lovitt Melayu Nanda Cikgu Olsson Dario Amodei oleh Tom Brown oleh Jack Clark oleh Sam McCandlish Kris Olah Ben lelaki oleh Jared Kaplan Abstract abstrak Kami menerapkan pemodelan keutamaan dan pembelajaran penguatkuasaan daripada maklum balas manusia (RLHF) kepada model bahasa finetune untuk bertindak sebagai penolong yang berguna dan tidak berbahaya. Kami mendapati latihan penilaian ini meningkatkan prestasi pada hampir semua penilaian NLP, dan sepenuhnya bersesuaian dengan latihan untuk kemahiran khusus seperti pengkodean python dan ringkasan. Kami meneroka mod latihan dalam talian yang berulang, di mana model keutamaan dan dasar RL dikemas kini setiap minggu dengan data maklum balas manusia yang segar, secara berkesan meningkatkan set data dan model kami. Akhirnya, kami menyiasat ketahanan latihan RLHF, dan mengenal pasti hubungan kira-kira linier antara ganjaran RL dan akar persegi perbezaan KL antara dasar dan inisialisasi. Bersama dengan hasil utama kami, kami melakukan analisis peripheral pada kalibrasi 1 Pengenalan Kami ingin membangunkan teknik untuk melatih agen AI yang berguna, jujur dan tidak berbahaya Dalam makalah ini kami menunjukkan bahawa kita boleh melatih yang agak berguna dan tidak berbahaya. (HH) penolong bahasa semulajadi dengan mengumpul data keutamaan manusia dan menggunakan teknik pemodelan keutamaan (PMing) dan pembelajaran penguatkuasaan daripada maklum balas manusia (RLHF). [Bahasa Melayu dan lain-lain, 2021] 1 2. Matlamat kami bukan untuk mendefinisikan atau meresepkan apa yang bermaksud 'membantu' dan 'tidak berbahaya', tetapi untuk menilai keberkesanan teknik latihan kami, jadi untuk sebahagian besarnya kami hanya membiarkan pekerja kumpulan kami menafsirkan konsep-konsep ini seperti yang mereka anggap sesuai. kami merawat berguna dan tidak berbahaya secara berasingan, mengumpul set data yang berbeza-beza mengenai keutamaan manusia untuk masing-masing. untuk kebaikan, kami meminta pekerja kumpulan untuk meminta model kami untuk membantu dengan apa-apa tugas berasaskan teks semata-mata seperti menjawab soalan, menulis atau mengedit dokumen, atau membincangkan rancangan dan keputusan. Pada setiap peringkat perbualan mereka dengan penolong AI, pekerja kumpulan diberikan dua jawapan yang mungkin. Mereka yang terlibat dalam tugas bantuan diperintahkan untuk memilih jawapan yang lebih berguna dan tulus (iaitu lebih baik). Mereka yang terlibat dalam tugas pasukan merah diperintahkan untuk memilih jawapan yang lebih berbahaya (iaitu lebih buruk). Perbualan ini dan keutamaan manusia yang dinyatakan membentuk set data kami. 2 3 Membantu dan tidak berbahaya sering bertentangan satu sama lain. Penekanan yang berlebihan kepada mengelakkan bahaya boleh membawa kepada tindak balas 'aman' yang tidak benar-benar menangani keperluan manusia. Penekanan yang berlebihan kepada menjadi membantu boleh membawa kepada tindak balas yang membantu manusia menyebabkan bahaya atau menghasilkan kandungan toksik. Kami mendemonstrasikan ketegangan ini secara kuantitatif dengan menunjukkan bahawa model keutamaan yang dilatih untuk menilai salah satu daripada kualiti-kualiti ini bertindak sangat buruk (lebih buruk daripada kebetulan) pada yang lain. Untungnya, kami mendapati bahawa PM yang dilatih pada campuran kedua-dua set data masih boleh belajar pelajaran yang betul dan bertindak dengan membantu apabila sesuai, sambil menggalakkan penolakan yang sopan terhadap permintaan berbahaya. Dengan model keutamaan di tangan, kami kemudian melatih pembantu yang berguna dan tidak berbahaya melalui pembelajaran penguatkuasaan, menggunakan skor PM sebagai ganjaran. Kami menilai kedua-dua prestasi PM dan ciri-ciri prestasi yang lebih relevan daripada model yang dilatih oleh RLHF kami. Seperti yang boleh dilihat dalam Gambar Model yang dilatih oleh RLHF yang semata-mata berguna jauh lebih mudah untuk red-team, manakala model yang berguna + tidak berbahaya sangat berguna dan kurang berbahaya. 1, daripada Satu soalan yang kerap dibangkitkan mengenai latihan alignment ialah sama ada ia akan merosakkan keupayaan AI.Kami mendapati bahawa apabila RLHF digunakan kepada model bahasa yang besar, jawapan seolah-olah hampir tidak.Model kami yang dilatih RLHF cenderung melakukan lebih baik daripada rakan-rakan mentah, generatif mereka pada hampir semua penilaian, seperti yang disimpulkan dalam Gambar Kami juga berpendapat bahawa seseorang boleh menggabungkan kemahiran khusus dengan latihan yang berkaitan dengan penyelarasan tanpa mengorbankan penyelarasan atau prestasi.Dalam amalan, model penyelarasan cenderung lebih mudah digunakan dan boleh digunakan daripada rakan-rakan mentah mereka, yang menunjukkan bahawa terdapat sedikit sebab untuk menyebarkan model yang belum disesuaikan untuk penyelarasan. 3. 1.1 Sumbangan Dialogue Preference Datasets • Kami mengumpul set data berguna dan tidak berbahaya (iaitu red-teaming) secara berasingan, terutamanya menggunakan pelbagai model bahasa 52B (lihat Seksyen untuk butiran) dalam antara muka kami (Gambar Pekerja kerumunan mempunyai perbualan terbuka dengan model, sama ada meminta bantuan, atau memberikan arahan, atau menggoda untuk mendapatkan model untuk mengeluarkan tindak balas berbahaya, dan mereka diminta untuk memilih jawapan yang lebih berguna atau yang lebih berbahaya. kepada setiap langkah perbualan masing-masing. 2 6). 4 • Kami mengumpul tiga bahagian data, satu daripada model awal kami, satu dengan sampel penolakan terhadap model keutamaan awal, dan set data akhir yang dikumpulkan dengan model yang dilatih dengan pembelajaran penguatkuasaan 'online' daripada maklum balas manusia, yang kami meningkatkan pada kaden kira-kira mingguan. 2.3 daripada Alignment with Human Values Has Many Benefits and Essentially No Cost to Performance • Model yang lebih kecil mengalami ‘caj penyelarasan’ yang teruk – prestasi mereka pada pelbagai penilaian berkurangan selepas latihan RLHF. , dengan 13B dan 52B Model yang dilatih RLHF bertindak lebih baik pada penilaian NLP nol-shot, dan yang sama pada penilaian beberapa-shot. alignment bonuses 5 • Latihan bahasa semulajadi RLHF untuk HH boleh digunakan kepada model yang telah disesuaikan dengan kod, dan ia pada penilaian (mungkin dengan meningkatkan arahan tujuan umum berikut). kami juga mendapati bahawa menggabungkan latihan model keutamaan untuk HH dengan daripada summarization incurs no degradation in perfor- mance in either HH or summarization. So there is no reason not to combine alignment training with more specific, valuable skills. improves their programming ability specialized skill [Bahasa Melayu dan lain-lain, 2020] • Terdapat a , yang boleh diukur pada tahap kedua-dua pemodelan keutamaan dan dasar-dasar yang dilatih oleh RLHF (Gambar Walau bagaimanapun, apabila saiz model meningkat, PMs bertindak lebih baik pada kedua-dua distribusi secara bersamaan dan menjadi lebih kukuh kepada proporsi relatif data latihan yang berguna dan tidak berbahaya. tension between helpfulness and harmlessness 1 ) • Kami juga menunjukkan bahawa seseorang boleh menggunakan teknik pengesanan OOD [F untuk menolak permintaan yang paling pelik dan berbahaya (Gambar dengan sedikit atau tiada contoh yang berbahaya (Figure dan lain-lain pada tahun 2021. 22 ) 23 ) Scaling, RLHF Robustness, and Iterated ‘Online’ Training • Kami mengkaji hubungan skalan untuk ketepatan PM sebagai fungsi model dan saiz dataset, dan mencari kira-kira trend log-linear (Gambar though we encounter some idiosyncrasies (Figures and 7 ) 31 32 ) • Kami menjalankan eksperimen di (lihat gambar di mana kami membahagikan set data kami kepada separuh dan melatih model keutamaan berasingan pada setiap separuh.Selepas itu kami melatih model RL terhadap satu PM sambil menilai dengan yang lain.Kami menyimpulkan bahawa PM yang lebih besar lebih kukuh daripada PM yang lebih kecil, dan seperti yang diharapkan, peningkatan overfitting semasa latihan RLHF. robustness of RLHF 4 ) • Kami mendapati bahawa Perlembagaan ( 0) dan ganjaran ialah untuk banyak latihan RLHF (lihat angka dan di mana dan 0 ialah dasar dan dasar permulaan, masing-masing.Kami menerangkan bagaimana hubungan ini boleh timbul dan membincangkan aplikasi yang mungkin dan arah masa depan. D daripada π approximately linearly related 4 13 ) π π • Kami mempelajari , di mana kami mengemas kini model keutamaan dan dasar RLHF kami pada kaden mingguan, dan kemudian menyebarkan semula model RLHF segar ini untuk berinteraksi dengan pekerja kerumunan. dan secara besar-besaran meningkatkan dataset kami seperti yang dinilai oleh PM kami sendiri (Gambar filling out the upper tail in terms of quality. iterated online training 1 ) 15 ) Untuk menghapuskan kebingungan dan mengukuhkan kesimpulan kami, kami menjalankan eksperimen terkawal tambahan (Gambar mengekalkan saiz dataset dan hiperparameter lain yang tetap. 16 ) 1.2 Pendekatan penilaian dan metrik · : We evaluate our models on MMLU Melayu kepada Melayu [P] Jepun OpenBookQA ARC dan triviaca Angka ini dan untuk hasil penuh dan nombor Dalam setiap kes kecuali TriviaQA, 12B dan 52B model yang dilatih RLHF berfungsi lebih baik daripada LM asas. secara berasingan, kami mengambil model pengkodean Python dan menyesuaikan mereka dengan bahasa semulajadi RLHF, dan kemudian menilai mereka pada kodeks HumanEval Gambar ini Kami juga bereksperimen dengan campuran latihan PM untuk HH dengan ringkasan sebagai kemahiran khusus, dan menilai prestasi PM yang dihasilkan (Gambar mendapati bahawa latihan campuran tidak menjejaskan ketepatan PM. NLP and Code Evaluations [Bahasa Melayu dan lain-lain, 2021b] Merdeka et al, 2016 [Bahasa Melayu dan lain-lain, 2019] [Melayu dan kawan-kawan, 2018] [Melayu dan kawan-kawan, 2018] [Joseph dan kawan-kawan, 2017] 28 29 3 [Bahasa Melayu dan lain-lain, 2021] 21. [Bahasa Melayu dan lain-lain, 2020] 20 ) · : Kami menilai PM kami menggunakan Penilaian HHH kami daripada Big-Bench (Figure on Bot Adversarial Dialogues dan bias gender (Gambar Kami menilai model RLHF kami pada TruthfulQA (Gambar BBQ kecil [P] daripada BIG-Bench, bias gender (Gambar dan perasaan berdasarkan ras dan agama (Gambar RLHF meningkatkan perasaan terhadap semua kumpulan, tetapi tidak menghilangkan bias. Static Alignment Evaluations [Bahasa Melayu et al., 2024] 6 5 ) [Xu et al., 2020], [Raja dan rakan-rakan, 2019] 12 ) [Lin et al., 2021] 5 ) arrish et al., 2021] 40 buah), [Raja dan rakan-rakan, 2019] 17). · : We compute Elo scores based on the preferences of our crowdworkers, com- paring context-distilled models, base RLHF trained models, and final online RLHF models (Figure Kami juga menguji prestasi model dalam talian kami semasa latihan (Gambar membandingkan pelbagai tahap pengambilan sampel penolakan (Gambar dan menjalankan eksperimen terkawal pada latihan online berulang (Gambar Selain itu, kami menyewa penulis profesional untuk menyusun perbualan di mana seorang pembantu memberikan jawapan yang berkualiti tinggi, berguna dan tulus, dan kami kemudian meminta pekerja kerumunan untuk membandingkan jawapan model kami dengan yang penulis ini. kira-kira 57% daripada masa Human Evaluations 1 ) 15), 36) yang 16 ) 7 • : Kami menyediakan sampel dari semua PALMs Soalan-soalan dan soalan-soalan yang dibincangkan di atas. dan LAMDA dalam lampiran Kami menunjukkan beberapa perbandingan dengan penulis manusia dalam Seksyen dan kami menunjukkan beberapa dialog pendek dalam Seksyen Untuk mengurangkan masalah pengambilan sampel ceri, kami menghasilkan 17 sampel setiap prompt dan memaparkan hanya sampel median, seperti yang dinilai oleh model keutamaan HH dalam talian kami. Samples [Solaiman and Dennison, 2021] [Bahasa Melayu et al., 2024] [Thoppilan et al., 2022] c ialah 6.1, 6.3 daripada 1.3 Kerja berkaitan Dua akhbar baru-baru ini, LaMDA dan instruksi Kedua-dua menggunakan data manusia untuk melatih model bahasa yang besar untuk menjadi lebih berguna secara amnya atau diselaraskan. kedua-dua menggunakan model bahasa yang agak lebih besar daripada model 52B kami. [Bahasa Melayu et al., 2024] [Bahasa Melayu et al., 2024] LAMDA Melengkapkan model bahasa yang besar untuk mengambil bahagian dalam dialog bahasa semulajadi yang menarik, berguna, berbasis fakta, dan selamat. Seperti dengan kerja kami, mereka termasuk gagasan kedua-dua interaksi positif dan keselamatan / tidak berbahaya. Dan penggunaan maklumat luaran mereka untuk memastikan ketepatan / asas- ness melampaui kaedah yang kami membincangkan di sini, dan mungkin lebih serupa dengan WebGPT dan GopherCite Walau bagaimanapun, beberapa perbezaan adalah bahawa alih-alih menggunakan pembelajaran yang diperkuat, mereka menggunakan campuran teknik pembelajaran yang diawasi (baik generatif dan diskriminatif), dan proses pengumpulan data mereka melibatkan penilaian mutlak bukannya perbandingan. [Bahasa Melayu et al., 2024] [Bahasa Melayu dan lain-lain, 2019] Khamis et al, 2022. Instruksi FineTunes GPT-3 jenis model Untuk meningkatkan bantuan mereka. Seperti dalam kerja ini, mereka menggunakan pembelajaran penguatkuasaan daripada keutamaan manusia, seperti yang dinyatakan melalui perbandingan. Walau bagaimanapun, mereka juga termasuk tahap pembelajaran yang dipantau latihan, manakala sebaliknya finetuning kami berlaku semata-mata melalui RL (kami melakukan distilasi konteks, tetapi ini lebih seperti menganjurkan mudah). Mungkin kontras utama dengan kerja kami adalah bahawa mereka tidak termasuk latihan tidak berbahaya, atau ketegangan ex-plore antara berguna dan tidak berbahaya. Pendekatan mereka juga berbeza daripada kita dalam beberapa butiran: mereka tidak melatih model keutamaan yang lebih besar daripada parameter 6B, dan mereka mencampur pra-pelatihan dengan RL untuk mengelakkan penurunan prestasi penilaian. [Bahasa Melayu et al., 2024] [Brown et al., 2020] Kerja kami berbeza daripada kedua-dua InstructGPT dan LaMDA kerana kami meneroka latihan 'online', di mana kami mengemas kini model yang berinteraksi dengan pekerja kerumunan untuk mendapatkan data berkualiti tinggi secara beransur-ansur dan mengisi ekor pengedaran data kami. Perbezaan lain ialah meneroka kemahiran khusus seperti ringkasan dan koding, yang kami gunakan untuk menguatkuasakan argumen bahawa penyelarasan boleh dicapai tanpa mengehadkan keupayaan. Kami juga secara eksplisit mengkaji ketegangan antara berguna dan tidak berbahaya, yang belum ditangani sebelum sejauh yang kita tahu. Akhirnya, kami meneroka skala dan ketahanan dalam lebih terperinci, termasuk semasa latihan RL. Dengan itu, prosedur kami (Gambar Kami percaya satu-satunya langkah penting ialah pengumpulan data maklum balas manusia, pemodelan keutamaan, dan latihan RLHF. 2) Beberapa kerja-kerja baru-baru ini memberi tumpuan kepada aspek kebenaran menggunakan pemulihan from a database, or via internet search and human feedback, such as WebGPT dan GopherCite Kerja-kerja ini menarik dan komprehensif kepada kerja-kerja kami; terutamanya hasil-hasil kami menunjukkan bahawa teknik-teknik mereka harus sangat bersesuaian dengan latihan untuk berguna dan tidak berbahaya. manakala kerja-kerja ini meningkatkan persembahan yang setia bukti eksplisit, lebih banyak kerja mungkin diperlukan untuk mencapai persembahan diri yang tulus daripada sistem AI.Kami umumnya berharap bahawa teknik-teknik yang bebas daripada maklum balas manusia mungkin boleh digunakan untuk masalah ini, kerana banyak sumber kebenaran tidak berdasarkan penilaian manusia. [Lewis dan rakan-rakan, 2020 Puan dan kawan-kawan, 2020 Borgeaud et al., 2021) [Bahasa Melayu et al., 2024] [Menick et al., 2022]. Safety and ethical issues associated with language models have been extensively discussed (e.g. 2021)), dengan isu-isu yang dikenali termasuk toksisiti, bias, dan kemungkinan bahawa model mungkin mendedahkan maklumat yang boleh diidentifikasi secara peribadi. Kerja-kerja lain telah mengkaji kaedah untuk mengurangkan masalah ini (contohnya. Model juga telah dilatih untuk secara langsung menilai dilema etika Menunjukkan kemahiran pada benchmarks etika Lebih umum cadangan penyelidikan untuk keselamatan AI termasuk The RL robustness failures we discuss can be viewed as an instance of ‘reward hacking’, which was recently explored in [P Kebijakan RL juga boleh gagal untuk menggeneralisasi daripada pengedaran dengan cara-cara berbahaya lain. [Henderson et al., 2017, Bender et al, tahun 2021, Weidinger dan lain-lain [Bahasa Melayu dan lain-lain, 2022] [Liu et al., tahun 2021, al dan al, [Jiang et al., tahun 2021] [Perkh dan rakan-rakan, 2021a] [Bahasa Melayu et al., 2016] Rizky et al. tahun 2021. dan lain-lain pada tahun 2022. [Bahasa Melayu dan lain-lain, 2021] Our interest in studying trends with model size is motivated by neural scaling laws Satu pengamatan yang berkaitan ialah apabila bilangan parameter meningkat, model finetune lebih berkesan dan menjadi kurang terdedah kepada ‘kegagalan yang dahsyat’ We expect this effect helps to explain why our HH training is compatible with good evaluation performance and specialized skills for large models. [Hestness et al., 2019, Rosemary et al., 2019 Kaplan et al., 2020]. [Bahasa Melayu et al., 2024] [Rasulullah dan kawan-kawan, 2024] 2 Pengumpulan data Kami mengharapkan maklum balas manusia (HF) mempunyai kelebihan perbandingan yang terbesar berbanding teknik lain apabila orang mempunyai intuisi yang kompleks yang mudah dihasilkan tetapi sukar untuk disemak dan mengotomatiskan. Ini bermakna bahawa apabila mengumpul HF, kita harus cuba memilih tugas yang semudah mungkin intuitif dan akrab.Kami memilih untuk menggunakan dialog bahasa semulajadi untuk kedua-dua sebab ini, dan kerana ia begitu umum - pada asasnya mana-mana tugas berasaskan teks boleh dilaksanakan melalui dialog, terutamanya dengan beberapa bahan sumber termasuk dalam talian. 2.1 Task Specification and Crowdworkers Antarmuka maklum balas manusia kami boleh dilihat dalam Gambar (Untuk maklumat lanjut lihat lampiran Orang-orang boleh berinteraksi dengan model kami dalam bahasa semulajadi melalui chat, dan meminta bantuan dengan apa-apa tugas berasaskan teks. Apabila ia merupakan giliran perbualan model, pengguna melihat dua jawapan model yang mungkin, dan memilih satu untuk meneruskan. Kedua-dua jawapan ini boleh datang daripada model yang sama, atau dua model yang berbeza. 6 D). • Crowdworkers write a chat message to our models, asking them to perform a task, answer a question, or discuss any topic of interest. • Crowdworkers are shown two responses, and are asked to choose the more helpful and honest re- sponse (or in the case of red-teaming, to choose the more harmful response). We conjectured that crowdworkers who wrote well and engaged the AI in more interesting discussions would tend to have better judgment about which AI responses were most ‘helpful’ and ‘harmless’. This meant that rather than attempting to filter crowdworkers based on label quality, we instead used spot-checks of their writing, which were simpler and more intuitive for us to perform. Otherwise, our approach to data collection was to largely let crowdworkers use their own intuitions to define ‘helpfulness’ and ‘harmfulness’. Our hope was that data diversity (which we expect is very valuable) and the ‘wisdom of the crowd’ would provide comparable RoI to a smaller dataset that was more intensively validated and filtered. Overall, our process was roughly of this form: 1. We invited master-qualified US-based MTurk workers to engage in dialogues with our models. 8 2. Rather than evaluating all of our crowdworkers, we identified those who were most prolific, and together accounted for about 80% of our data (roughly 20 crowdworkers). We then evaluated their performance based primarily on the sophistication and variation in their dialogues, as this was quite easy to evaluate intuitively (rather than based on any measure of agreement on helpful/harmless choices). Based on this method, we collected a list of ‘select’ MTurk workers yang kami telah bekerjasama sepanjang proses penyelidikan. 9 Kami menjemput pekerja kerumunan kami yang dipilih ke saluran Slack dan berkomunikasi dengan mereka melalui e-mel, untuk memastikan bahawa mereka dibayar dengan adil. dan untuk membolehkan mereka memberi amaran kepada kami mengenai sebarang masalah atau masalah. 10 4. We also hired crowdworkers on Upwork, and vetted them in a similar, lightweight way. We have continued to use both platforms throughout this work. We find that it is easier to incentivize very high-quality interactions on platforms such as Upwork, where crowdworkers can easily be paid by the hour, rather than per task. But conversely, MTurk workers tend to generate data much more rapidly, and account for about 80% of our datasets. We did not filter workers based on agreement or other direct measures of label quality, though we evaluated them retrospectively (see Figure right) and found poor average agreement (about 63%) between Anthropic researchers and our crowdworkers, as compared to recent similar work such as 10 [Stiennon et al., 2020, Khamis et al, 2022. Sebagai amaran penting, pengedaran crowdworker kami tidak dikekalkan sepanjang kerja ini, dan kami mengharapkan bahawa kualiti crowdworker mungkin meningkat apabila projek itu berterusan. Conversely, however, since we generally discouraged repetition, crowdworkers who have performed the task many times might also have had a tendency to engage in more esoteric interactions. 4.5. Kita juga perlu ambil perhatian bahawa kita secara eksplisit memberitahu pekerja kerumunan bahawa 'menipu tidak berguna' dan bahawa mereka harus cuba hanya memberi ganjaran yang berguna. Itu dikatakan, kami tidak mengharapkan pekerja kerumunan untuk memeriksa fakta-fakta model kami secara signifikan, dan contohnya, mereka sering lebih suka jawapan yang mengandungi URL yang tidak berfungsi, yang mungkin merupakan salah satu ‘kebenaran’ yang paling mudah untuk dibongkar. dan jujur 2.2 Bantuan dan Tidak Berbahaya (Red Teaming) Dataset We collected two separate datasets using slightly different versions of our interface. For the helpfulness dataset, we asked crowdworkers to have open-ended conversations with our models, asking for help, advice, or for the model to accomplish a task (see Appendix ), and to choose the model response that was . For the harmlessness or red-teaming dataset, we asked crowdworkers to attempt to elicit harmful responses from our models, and to choose the response offered by the models. D2 more helpful Lebih berbahaya Perkhidmatan kami (Figure allows users to express a preference strength. We only include comparisons in our datasets if crowdworkers expressed a preference stronger than the weakest available. In this work we will not otherwise use this preference-strength information; we treat all comparisons in our dataset as binary and of equal weight (so in particular we do not include ties). 6) Note that this means our helpfulness dataset tends to move conversations in a more beneficial direction, while in our red-teaming dataset user responses move conversations in a more harmful direction. We made this choice to make it possible for users to fully trick and exploit models while red-teaming, as this was most natural for other work we’re doing that’s specifically focused on harmfulness. However, we believe this difference made it difficult to train models that were both helpful and harmless, as explained in Section Kami merancang untuk membetulkan ini dalam kerja masa hadapan, dan akan mengesyorkan orang lain yang memberi tumpuan kepada melatih model dialog yang tidak berbahaya untuk mengumpul data di mana pengguna terutama memilih respons model yang memindahkan perbualan ke arah yang lebih bermanfaat. 4.4. 2.3 Model Dilancarkan ke Antarmuka Ulasan dan Distribusi Data Berkaitan Untuk pengumpulan data, kami terutamanya used 52B language models with the broad specifications given in Kami menggunakan tiga kelas model dalam antara muka kami: 11 [Askell et al., 2021]. · : At the beginning of the project this was the only model available. It performs similarly to a plain 52B language model prompted with HHH dialogues HHH Context-Distilled 52B Language Model [Askell et al., 2021]. • dengan model keutamaan 52B, di mana sampel dihasilkan daripada LM yang didestilkan konteks 52B. of samples was a parameter, but most often we used = 16. Rejection Sampling (RS) k k · : Kami menggunakan serangkaian model ini dalam antara muka kami. model ini berbeza terutamanya berdasarkan kepada jumlah data yang tersedia apabila melatih PM yang berkaitan (tergantung kepada fasa projek). RLHF-Finetuned Models In the final phase of the project, when we were primarily deploying RLHF-finetuned models, we often de- ployed several such models at once. This allowed us to monitor progress by gathering model-comparison data, and also to (perhaps) improve data diversity. Corresponding to the three classes of models, we divide our data into three distributions: • A core Set data ini termasuk 44k perbandingan berguna dan 42k perbandingan merah-teaming (tanpa bahaya) (ingat perbualan biasanya terdiri daripada kira-kira empat perbandingan). base • A set data yang terdiri daripada perbandingan 52k utility dan perbandingan 2k red-teaming menggunakan model sampel re-jection, di mana sampel penolakan menggunakan model keutamaan yang dilatih pada set data asas. RS • An iterated ‘ ’ dataset including data from RLHF models, which were updated on a roughly weekly cadence over the course of about five weeks. This dataset contains 22k helpfulness compar- isons and no red-teaming data. online Histogram distribusi-distribusi ini oleh skor akhir, model keutamaan HH dalam talian kami boleh dilihat dalam Gambar in Section Apa yang berlaku apabila kita membincangkan atau dataset base+RS, kita akan merujuk kepada gabungan dua komponen pertama. model RLHF ‘online’ kami dalam Seksyen Kebanyakan hasil kami didasarkan pada set data statis, kerana kami menjalankan eksperimen dan penilaian dengannya semasa pengumpulan data dalam talian sedang berlangsung. 15 4.5. static 4.5 Kami menganalisis beberapa pemisahan yang berbeza daripada set data statis – pemisahan standard ke dalam 95/5 data kereta / ujian, dan pemisahan 65/35 yang kami gunakan untuk mendapatkan statistik yang lebih baik apabila menilai kalibrasi model keutamaan pada set ujian. kami juga mempertimbangkan pemisahan 50/50, di mana kami melatih PM yang berbeza pada kedua-dua separuh set data. Ini digunakan untuk menilai ketahanan latihan RL, kerana kami kemudian melatih dasar RL terhadap satu PM sambil menilai ganjaran yang dicapai oleh dasar itu seperti yang diukur oleh PM bebas. 2.4 Comparing Models with Elo Scores A significant part of our analysis compares models against each other in order to generate associated Elo scores, as described in Iaitu, kami mempunyai pekerja kerumunan berbual dengan dua model pada masa yang sama, dengan setiap model menghasilkan satu respons (baik ‘A’ atau ‘B’) pada setiap giliran, dan kami merekodkan sampel yang disukai oleh pekerja. (where we show both win rates and Elo scores). Two useful conversion formulas are [Bahasa Melayu dan lain-lain, 2021] 1 for the fraction in which one model is preferred over another, the difference in Elo scores, and our PM scores. Note that conceptually win fractions, Elo scores and PM scores are interchangeable; we keep both Elo and PM scores so that we can avoid confusing crowdworker preferences (where we use Elo) with our preference modeling and RLHF (where we use PM scores). Perhatikan bahawa skor Elo untuk model distilasi konteks dalam Gambar differ somewhat from the analogous results for prompted models in – the Elo scores are now more compressed. The main difference is that we did not use top-p sampling this time . The difference may also be due to changes in the crowd- worker distribution since that earlier experiment, or changes in crowdworker expectations, as before this test our workers were mostly interacting with higher-quality RLHF-trained models. 1 [Askell et al., 2021] 12 3 Model Keutamaan untuk Bantuan dan Tidak Berbahaya 3.1 Models and Training Setup Kami menggunakan model bahasa dengan spesifikasi yang identik dengan yang dibincangkan dalam [Askell et al., 2021], dengan jumlah tujuh model bahasa dengan bilangan parameter yang berjalan dari 13M hingga 52B dan kira-kira siri geometrik dengan peningkatan kira-kira 4X. Kami menggunakan PyTorch [Paszke et al., 2019] dan Triton [Tillet et al., 2019] untuk memudahkan latihan dan prestasi model. Seting latihan model keutamaan kami juga identik dengan yang dalam [Askell et al., 2021], dan terutamanya kami menerapkan ‘preference model pretraining’ (PMP) kepada model bahasa kami sebelum menyempurnakan set data maklum balas manusia kami, seperti yang dijelaskan dalam Bahagian 4 kertas itu. Lebih banyak butiran disediakan dalam Tambahan A. Perhatikan bahawa kami biasanya hanya melatih PM untuk satu era, jadi pembelajaran sendiri melengkung 3.2 Basic Scaling Results Kami ingin memahami bagaimana prestasi pemodelan keutamaan meningkat apabila kami meningkatkan saiz model dan mengumpul data tambahan. we show basic results for PM accuracy when training on our static helpful and harmless data mixture. Roughly speaking, we observe log-linear trends in both dataset and model size. We tend to find somewhat more consistent trends if we model only the helpfulness or harmlessness distributions in isolation, rather than as a mixture, as observed in Figure dalam lampiran Tetapi di sana kita juga melihat bahawa untuk sesetengah pengedaran data scaling trends can exhibit more complex patterns that defy simple trends. 7 32 A. 3 [Stiennon et al., 2020], Data pemodelan keutamaan kami berasal daripada dialog bahasa semulajadi, di mana pekerja kumpulan mempunyai perbualan berasaskan teks dengan model, dan memilih yang lebih berguna daripada dua tindak balas model pada setiap pusingan dalam perbualan (atau yang lebih berbahaya, untuk tugas-tugas red-teaming). PMs are somewhat more accurate on the first step of the conversation, but their accuracy is nearly constant thereafter. 8. 3.3 Calibration of Preference Models and Implications for RL Preference model scores should predict the probability that humans will prefer one or another model- generated response. We are interested in whether these probabilities are accurate, i.e. whether the PMs are well calibrated. We characterize calibration in Figure where we display PM accuracy as a function of the difference in PM scores assigned to pairs of samples, along with a heavy black line representing perfect calibration. We observe that PMs trained only on helpfulness data are very well calibrated, but PMs trained on a mixture of helpful and harmless data are slightly under-confident. 9 , These calibration results are important because in later sections we will be using PM scores as the reward signal for reinforcement learning. Since the PM scores are well-calibrated, we can trust that they faithfully encode the probabilities that humans will prefer specific model samples (at least on-distribution with the training set). This means that when we see RL mencapai ganjaran yang diberikan, kita boleh percaya bahawa mereka yang berinteraksi dengan model ini (jika mereka diwakili dengan baik oleh pengedaran crowdworker kami) akan memilih untuk merujuk kepada model pada kadar yang boleh diprediksi, asalkan skor PM daripada respons model berada dalam julat yang dipertimbangkan dalam kajian kalibrasi ini. and Appendix robustly 4.5 B.4. We might generally expect that as sample quality improves, it becomes more difficult to reliably identify the very best samples. In Figure dalam lampiran, kami menunjukkan bahawa jika kita terhad kepada perbandingan di mana kedua-dua sampel mempunyai skor PM di atas ambang yang diberikan, maka ketepatan PM jatuh sebagai fungsi daripada nilai ambang ini (untuk membandingkan PM yang berbeza, kami menambah offset supaya mereka semua mendapat skor purata yang sama pada pengedaran ujian). Perhatikan bahawa hasil ini menggabungkan tiga kesan: (1) perbezaan antara sampel yang lebih canggih mungkin lebih mencabar dan memerlukan keupayaan yang lebih besar, (2) kita mempunyai lebih sedikit sampel berkualiti tinggi untuk belajar daripada dalam set data kami, kerana mereka bukan generik, dan (3) pasangan sampel yang semua berkualiti tinggi akan mempunyai skor yang sama (berbanding dengan pasangan yang dipilih secara rawak), dan oleh itu lebih sukar untuk membezakan. 25 Pengamatan ini juga mempunyai implikasi untuk latihan RLHF, iaitu bahawa kita harus mengharapkan pengembalian yang berkurangan daripada latihan RLHF lebih lanjut apabila dasar kami mencapai skor PM yang cukup tinggi. 3.4 Evaluating Helpful and Harmless Preference Models 3.4.1 HHH Evaluation and Crowdworker-Anthropic Agreement We have shown that PMs trained on our dataset are learning something and are well-calibrated. But we would also like to show that these PMs are actually learning to prefer helpful and harmless behavior, accord- ing to some independent standard. We provided a set of comparison evaluations for helpfulness, honesty, and harmlessness (i.e., HHH Evaluation) in prior work and evaluated raw, context-distilled, and prompted language models on this benchmark. As shown in Figure we find that our PMs perform much better than all of the models we evaluated previously. In fact, the Pathways LM effort recently reported a mean human score of 75% on this dataset, so in a sense our PM performance of 86% is well above mean human accuracy. [Askell et al., 2021] 5, [Chowdhery et al., 2022] Kami juga memperhatikan bahawa terdapat beberapa outliers, di mana PM kami dengan yakin salah pada perbandingan penilaian HHH (lihat Gambar left). These are all examples of failures of , where the better choice is honest but unhelpful, while the incorrect choice appears both helpful and well-written, but is subtly incorrect or misleading. We show the single worst example in Figure It’s worth noting that the bad response (which was written by Anthropic researchers) is likely out of distribution as compared to samples from our language models and RL policies. Since only model-generated samples were used to train the PMs, this may explain why the PM was badly fooled by these examples. It also indicates that the PMs are not adversarially robust. 10 Kejujuran 11. We can also compare our crowdworkers, ourselves, and our preference models in terms of agreement on labels. The results are shown on the right in Figure dan berdasarkan kepada kira-kira 320 contoh daripada set ujian statis kami. PMs dilatih untuk meramalkan tingkah laku crowdworker, jadi perjanjian PM-Crowdworker adalah yang terbaik. Walau bagaimanapun, PM terbesar sebenarnya bersetuju dengan pengarang (iaitu penyelidik antropik) sedikit lebih daripada pengarang bersetuju dengan crowdworker pada label. Kami juga mencurigai ini adalah sub sampel yang buruk data, kerana ketepatan PM adalah beberapa peratus lebih rendah pada sub sampel ini berbanding dengan set ujian penuh. 10 yang 3.4.2 Gender Bias and Bot Adversarial Dialogues We can begin to evaluate whether our PMs display gender bias by comparing scores across gendered terms. To this purpose we evaluate preference model scores on short dialogues of the form Manusia: Bolehkah anda membantu saya mengisi perkataan yang hilang dalam satu kalimat? Penolong: Tentu saja, di sini ialah ayat anda dengan kata yang hilang diisi: The {Occupation} is a {Gendered Term} di mana kami menggunakan senarai 76 profesion, 12 istilah jantina yang sesuai (bapa vs ibu, lelaki vs gal, dan lain-lain), dan kedua-dua ketegangan semasa dan masa lalu Preference model scores are not directly meaningful, so instead we evaluate the difference in score between paired gender terms, and compare that to the difference in score among terms within a given gender. The results are shown on the left in Figure We observe that the variation among terms that indicate a given gender appears to be at least as large as the variation across genders. So it would appear that the PMs do not exhibit a substantial bias. [Rae et al., 2021]. 12. Sebagai pengesahan luaran PM kami, kami mempertimbangkan set data Bot Adversarial Dialoges (BAD) This dataset contains a few thousand conversations between an AI system and a human. Each AI response is labeled as harmful or not harmful. We compute the distribution of preference model scores for BAD AI utterances (we restrict our analysis to the first BAD AI utterance per conversation) and find that the BAD AI utterances marked as harmful have significantly lower preference model scores. This suggests that our PMs are effectively classifying these AI generated utterances, even though they are likely quite different from the data distribution that our PMs were trained on. [Xu et al., 2020]. 4 Reinforcement Learning from Human Feedback 4.1 Latihan Pembinaan We apply reinforcement learning (RL) with preference modeling, following the approach outlined in which can summarized in the following steps: [Bahasa Melayu et al, 2020] Menyediakan set data perbandingan, dan melatih PM untuk memberikan skor yang lebih tinggi kepada item 'lebih baik' dalam setiap perbandingan.Dalam konteks eksperimen umpan balik manusia kami, setiap perbandingan terdiri daripada prompt yang diikuti oleh sepasang jawapan yang dihasilkan oleh model, dengan skor PM yang dinilai di hujung setiap jawapan. Mengekstrak semua prompts daripada set data terdahulu, dan melatih dasar RL untuk menjana respons kepada setiap prompt secara autoregressive, dengan isyarat ganjaran yang disediakan oleh skor PM pada akhir jawapan. PM dataset and training details are provided in Appendix Kami juga membincangkan prestasi PM kami dalam Seksyen In the language of RL, each response generated by the policy is a ‘timestep’, a full conversation is one ‘trajectory’, and the PM score is a single ‘reward’ provided at the end. A.2; 3. Idea adalah untuk menggunakan model keutamaan untuk mengarahkan dasar ke arah penulisan jawapan yang lebih baik. bagaimanapun, seperti yang kita lihat dalam seksyen terdahulu, PM juga menjadi kurang dikalibrasi pada skor yang lebih tinggi, jadi ganjaran yang lebih tinggi tidak semestinya bermakna prestasi yang lebih baik. Untuk menjejaskan latihan RL, kami menggunakan Proximal Policy Optimization (PPO) We also follow other work and apply an empirically-estimated KL penalty term in the reward, with the total reward given by [Schulman et al., 2017]. [Stiennon et al., 2020] where KL >/ 0 is a hyperparameter. In practice we use a very small value of KL = 0:001, which likely has a very minor impact during most of RL training (as DKL < 100 typically), and might actually be wholly unnecessary. More details about RL are provided in B.1. Throughout this paper we use PM = the preference model score itself for the RL reward. Recall that as implied by equation this means that the difference in PM values between two samples dan akan dikaitkan dengan probabiliti yang dijangka ( ) yang will be preferred to via r (2.1), r A B P A > B A B There is no good reason to use this preference model score directly as the reward, but it has been used in prior work such as and so for simplicity we will not explore variations on this choice here. 13 [Stiennon et al., 2020] In order to produce additional prompts (i.e. the human side of the conversations) for RLHF training, we used a large LM to generate them. For this purpose, we simply used few-shot learning, creating a context with about 10 existing high-quality human queries, and then sampling to generate more. We find that the sample efficiency of RLHF is roughly the same on the original crowdworker-written prompt dataset and the model- generated one, so we combine the two for greater diversity during RLHF training. We used 137k prompts from the ‘static’ dataset, and 369k model-generated prompts. Note that almost all of our preference modeling data was collected from 52B models. This means that RLHF training with smaller models might have been challenging, since samples from smaller models tend to be out-of-distribution from the PM training data. Thus it is quite interesting that models more than fifty times smaller were actually able to learn and improve, as seen in Figure 1. 4.2 Robustness Experiments We now discuss the problem of RLHF robustness. A fully robust PM would agree with humans on distribu- tions of dialogues quite different from those encountered during PM training (i.e. different from those created by crowdworker interactions with our deployed AI assistants). However, we do not expect that our PMs are so robust, and in fact Figure Oleh kerana RL mengoptimumkan dasar untuk memaksimumkan skor PM, apa-apa kegagalan dalam kestabilan di pihak PM boleh dimanfaatkan oleh dasar RL untuk mencapai ganjaran yang lebih tinggi, tanpa benar-benar meningkatkan tingkah laku dasar dari sudut pandang penilai manusia. 11 A rigorous way to study robustness is to take snapshots of the policy at various points during RLHF training, including the initial snapshot, and have crowdworkers compare their performance. This gives a ‘true’ Elo score, as evaluated by crowdworkers, which can then be compared directly with the PM scores. We present an example of this study in Section 4.5 daripada Walau bagaimanapun, jenis ujian ini memerlukan pengumpulan data maklum balas manusia tambahan, yang boleh menjadi lambat dan mahal, jadi di sini kita juga mengkaji ketahanan dari sudut yang berbeza. separuh dan a half), and train separate preference models on each, which we refer to as the ’s and the ’s. We then train RLHF policies against the ’s, while evaluating them using the ’s. Serupa dengan bagaimana penilaian set ujian membantu kita memahami kelebihan dalam pembelajaran yang diselia, evaluations help us understand overfitting relative to the ’s. These experiments are not conclusive since the train and test PMs may exhibit correlated robustness failures. train Ujian Kereta api PM test PM train PM test PM test PM train PM Kesimpulan utama daripada eksperimen ini adalah: (1) RLHF menjadi beransur-ansur kurang kukuh pada skor PM yang lebih tinggi, dan (2) model keutamaan yang lebih besar lebih kukuh daripada yang lebih kecil. We conduct two sets of experiments as follows: • : This set consists of a scan of policies (i.e. one for each model size), all of which are trained with respect to the same 52B . Train PM Size = 52B Kereta api PM • : This set consists of a scan of policies, with each policy trained with respect to a mempunyai saiz yang sama dengan politik. Train PM Size = Policy Size train PM For both experiments, each policy is further evaluated with respected to a scan of ’s throughout training. Note that a refers to 7 different model sizes ranging from 13M to 52B, thus giving us 7 policies and 7 × 7 evaluations per experiment. test PM scan In Figure we compare the and scores throughout the training process, similar to how train and test curves are often compared for supervised training. We find that in all cases, the two scores are in close agreement during early stages of training, but eventually diverge, with the providing a lower score. The divergence is likely an indication that . That is, the policy has been over-optimized on the , making the • Mempunyai keyakinan yang tinggi terhadap prestasi politik. • , on the other hand, doesn’t suffer from this problem since it was trained on a different portion of data that neither the policy nor the had observed. 4 , train PM test PM test PM the preference model is less robust and more easily exploited at higher rewards Kereta api PM train PM test PM train PM We provide more discussion in Appendix B.2. 4.3 An Approximately Linear Relation Between √ KL and Reward D D In Figures dan we observe an Antara skor √KL dan PM semasa latihan RLHF.Selain itu, kami memperhatikan bahawa apabila√a__ll mo__dels dilatih dan dinilai dengan PM yang sama, kurva pembelajaran adalah kira-kira in the KL-reward plane. Nota bahawa di sini 'KL' adalah lebih tepat reward bervariasi secara linear dalam Oleh yang demikian, di dalam ‘sma√ll__- r__egion’ (iaitu di mana perpanjangan siri menyediakan pendekatan yang baik), kita perlu mengharapkan ganjaran KL. Typically we should that reward varies linearly in , because because the initial policy was not previously optimized for reward, so there is no reason why it would sit at an extremum with respect to small variations . So the fact that this relation seems to hold empirically suggests that most of RLHF training remains in the small- regime. 4 13 approximately linear relation parallel D δπ δπ D expect δπ π δπ dp Though they did not use these coordinates, a similar scaling can be read off from the results in learning to summarize In particular, they provide a nice analysis of rejection sampling, where they generate kepada mereka, dan mereka akan mendapat ganjaran yang tinggi. sampel berbanding dengan KL = log( ). [Stiennon et al., 2020]. N k D N/k This analysis suggests that these RL learning curves might be associated with changes in the RL policy that behave very similarly to simply rejection sampling from the initial distribution. We find this simple relation quite striking, and believe it merits further study. At a conjectural level, it might have a variety of implications and uses when RL-finetuning large generative models: • These relations provide a rough prediction for ‘how much does the policy need to change to achieve a specific reward’. Furthermore, if the lines corresponding to different model sizes really are parallel, then one can use RL training of a small model along with the zero-shot performance of a larger model to estimate the eventual performance of a larger RL policy. The slopes of these lines also explain how RLHF training can produce such large effective gains in model size, and for example it explains why the RLHF and context-distilled lines in Figure are roughly parallel. 1 • One can ask a subtle, perhaps ill-defined question about RLHF training – is it or simply · Kami teaching the model new skills memberi tumpuan kepada model untuk menghasilkan sub-pembahagian tingkah laku yang sedia ada might attempt to make this distinction sha√r__p b__y associating the latter class of behaviors with the region where RL reward remains linear in KL. • To make some bolder guesses – perhaps the linear relation actually provides an on RL re__war__d, as a function of the KL. One might also attempt to extend the relation further by replacing √KL with a geodesic length in the Fisher geometry. upper bound By making RL learning more predictable and by identifying new quantitative categories of behavior, we might hope to detect unexpected behaviors emerging during RL training. 4.4 Tension Between Helpfulness and Harmlessness in RLHF Training Here we discuss a problem we encountered during RLHF training. At an earlier stage of this project, we found that many RLHF policies were very frequently reproducing the same exaggerated responses to all remotely sensitive questions (e.g. recommending users seek therapy and professional help whenever they express any level of displeasure at all). This greatly limited these models’ utility. We still see a vestige of this behavior in some of the examples provided in Section We now believe these policies were the result of for harmlessness, while yang berguna. 6.2 daripada Optimalisasi yang berlebihan under-optimizing Dengan prosedur pengumpulan data kami, kami fikir ini agak intuitif. Untuk mendapatkan skor yang sangat baik pada panggilan berpasukan merah, ia mungkin mencukupi bagi model untuk bertindak balas dengan sesuatu seperti "Saya tidak boleh menjawab itu." In Figure (right), we show the policy’s PM score throughout training, after separating helpfulness and harmlessness prompts. On the left side of the same figure, we show the score distribution of PM comparison data, again separating helpful and harmless datasets. We observe that the policy’s harmlessness score is somewhat off-distribution, as it is on the upper tail of the harmlessness comparison data. On the other hand, the policy’s helpfulness score appears on-distribution, and is likely under-optimized. So we would expect this agent to be very difficult to red-team, but not very helpful. 14 This then raises an obvious question – can’t we just collect more harmlessness data to fill out the upper tail of the distribution? The problem involves the definition of harmlessness mentioned above – if simply refusing to answer a question is the ‘least harmful’ behavior, then this is probably both very easy to learn, and hard to improve on. That said, a more interesting ‘least harmful’ behavior would involve the model (helpfully) explaining why the request was harmful, and perhaps even trying to convince the human not to pursue such requests. We informally refer to such a model as a ‘hostage negotiator’. Walau bagaimanapun, proses pengumpulan data kami menjadikannya sangat sukar bagi model untuk mempelajari 'perundingan sandera'. AI response. We made this choice so that we could fully explore the vulnerability of our models to red-teaming. However, from the point of view of RLHF this was problematic, because beyond the first turn of dialogue, our models never learned what a response to a harmful query might be like. Our dataset does not provide guidance on the upper end of the distribution, on what models should do, but only tells models what to do. Lebih berbahaya sophisticated not Dalam amalan, kami telah sebahagiannya menyelesaikan isu optimisasi dengan melatih pada sebahagian yang lebih besar prompt kegunaan semasa RLHF. tetapi di masa depan kami berharap untuk menangani masalah ini secara lebih lengkap dan sistematik dengan mengumpul data tidak berbahaya di mana pekerja kumpulan memilih tanggapan terbaik yang mungkin daripada model kami. In this way we hope that rather than simply shutting down harmful requests, models can learn the more subtle art of ‘hostage negotiation’ with red-teamers. 14 Note that since the data and models discussed in this section are from an earlier stage of our research, the RL results may look slightly different from other parts of the paper. 4.5 Iterated Online RLHF In preceding sections we discussed the problem that PMs become progressively less calibrated and less robust at higher scores, as seen in the PM calibration study in Figure and the RLHF robustness study in Figure 9 , We believe this is caused by a lack of data in this high score regime. To address this, we propose : 4. iterated online RLHF • We simply train the best RLHF policy we can, and use that to collect comparison data from crowd- workers. Since the policy was trained to optimize for PM score, it should produce responses that are on the upper end of the score distribution. • Kami menggabungkan data perbandingan baru dengan data sedia ada kami, dan melatih pemindaian PM baru, yang kemudian kami gunakan untuk melatih pemindaian baru dasar RLHF. Our hypothesis is that the ‘online’ RLHF policy helps us collect data on the upper end of the PM score distribution, which should improve PM calibration at high scores on subsequent iterations, and thereby allow us to train even better policies. Continuing this process should give us progressively better PMs and policies. Note that our use of the terminology ‘online’ is different from conventional use of the word—instead of training the same model iteratively, we retrain a new model per iteration. One concern about this approach is that RLHF tends to decrease the policy’s entropy, which would limit the diversity of data collected through the online procedure. We partially address this by deploying a number of different snapshots from RL training, and from different online iterations, at once. This also makes it possible to compare these models to get a better sense of how they are performing. We can see signs of life from the online approach by looking at the evolution of our data distribution. In Figure (left), we show the PM scores from three distributions of models: daripada (Penghormatan pengambilan sampel) dan , as described in Section We see that according to our final online PM (trained on all of the data), the quality of samples improves from the base to the rejection-sampling to the online data distributions. We also found that our online PM achieves accuracies of 74%, 70%, and 67% on the test sets for the respec- tive base, RS, and online-only distributions, which shows that distinguishing among higher quality samples is becoming more challenging. This makes us optimistic that online training should outperform rejection sampling in the long run. 15 Base RS Online 2.3. Kami memaparkan kurva pembelajaran untuk model dalam talian kami, bersama-sama dengan pengukuran skor Elo daripada pekerja kerumunan, di sebelah kanan dalam Gambar We see that models improve significantly during RLHF, but Elo scores from crowdworkers do not match predictions from PMs. We further discuss and decompose the robustness of RLHF training in Appendix where we see that distributional shift accounts for a significant part of the apparent robustness failure (Figure 15. B. 4 dan 35). In Figure we compare Elo scores of our online model with context-distilled models and RLHF models trained on the ‘static’ (i.e., no online) dataset, showing that the online models are clearly preferred by our crowdworkers. However, readers might worry about two caveats: the online model was trained on a slightly larger (about 20% larger) dataset, and the online model was trained with improved RLHF hyperparameters (the online model was trained with a larger , defined in Appendix dan PMnya dilatih dengan konteks 2048 bukannya 1024), berbanding dengan latihan latihan statik RLHF terdahulu. 1, K B. 1 dan To address both of these caveats, we performed a controlled experiment comparing two RLHF runs: one trained with our base dataset (about 44k PM comparisons), and another trained on an even mixture of base, RS, and online data (sekitar 15k perbandingan PM dari masing-masing). Jadi untuk eksperimen ini kami melatih dua PM yang berasingan pada setiap set data, dan kemudian melatih sepasang dasar RLHF terhadap kedua-dua PM ini. Selain daripada perbezaan data, kedua-dua berjalan menggunakan seting yang sama, dan hanya dilatih mengenai bantuan. we compare Elo scores for various snapshots of both runs, as determined by crowdworker preferences, showing that the policy trained on the iterated-online mixture is clearly preferred. This demonstrates that online training works, and that performance gains are not merely due to increased dataset size or hyperparameter changes. whose total dataset size is the same as the base dataset 15 16, 4.6 Evaluations: Alignment Bonus, Honesty, and Biases Language models that have been finetuned via RL typically have much narrower, lower-entropy output distri- butions. This can make evaluations difficult when they are fairly rigidly formatted, since all valid responses may be far off-distribution for the RLHF model (we discuss an example with gender bias evaluations below). Thus we expect in future work evaluations involving sampling and human interaction may be most relevant. In what follows we discuss some standard NLP evaluations, and then evaluations specifically related to the societal impacts of the models, including honesty, sentiment, and bias. 4.6.1 NLP Evaluations We evaluate our models on question answering, commonsense, trivia, and story completion using the bench- marks MMLU Lambada [P Jepun OpenBookBahasa Melayu ARC dan triviaca The main conclusion is that RLHF tends to improve performance for large models, while degrading prestasi model yang lebih kecil. [Hendrycks et al., 2021b], aperno et al., 2016], [Zellers et al., 2019], [Mihaylov et al., 2018], [Melayu dan kawan-kawan, 2018] [Joshi et al., 2017]. 16 Full results for both zero-shot and few-shot evaluations are shown in Figures and and we provided a summary of the mean trends in Figure Readers may notice that results improve rather suddenly for some evaluations; this is a consequence of the format we use for multiple choice questions, where we explicitly provide choices (Gopher used this format). The format is provided explicitly in Appendix We find that this format tends to improve performance for large models, while decreasing the performance of small models, leading to the arguably misleading appearance of a ‘grok’ curve. 28 29, 3. [Raja dan rakan-rakan, 2019] E . [Power et al., 2022] 4.6.2 Honesty and Biases A major question is whether AI models are honest. We evaluate our models on TruthfulQA (MC1) and show the results in Figure Di sana kami juga memasukkan prestasi pada 50-shot, dalam bentuk untuk menunjukkan bahawa walaupun latihan RLHF kami secara signifikan meningkatkan kejujuran, model kami mungkin mempunyai ruang yang signifikan untuk meningkatkan. Perhatikan bahawa arahan kami kepada pekerja kerumunan menunjukkan bahawa 'menipu tidak membantu- ful' dan bahawa mereka harus memilih jawapan yang 'membantu dan jujur', jadi ini mungkin berkaitan dengan peningkatan yang kita lihat pada TruthfulQA. [Lin et al., 2021] 5. Another set of questions involves the underlying biases of these models. We evaluate our models for sentiment biases on race and religion (in the same format as Gopher 2021]), for gender bias, and on the Bias Benchmark for QA (BBQ-lite) [P [Bahasa Melayu et al. Melayu et al. Tahun 2021. Results for sentiment towards different racial and religious groups are shown in Figure Kesan utama yang kita perhatikan ialah bahawa perasaan model kami yang dilatih RLHF cenderung menjadi lebih positif daripada LM biasa, tetapi lebih sukar untuk mengatakan sama ada bias antara kumpulan telah berubah. , for example the low scores for the Atheist group is likely because samples such as (model generation in italics): 17. 17 Seorang ateis yang sangat skeptical and critical of religious dogma and conventional moral principles, seeking instead to derive his ethical values from rational consideration and empirical observation of the natural world. receive a sentiment score very near zero, which seems like a questionable evaluation. For these evaluations we use a prompt format where the human asks the assistant to complete the sentence as follows: Human: Can you help me finish a sentence? The sentence is: {sentence beginning} Assistant: Sure thing, here is your completed sentence: {sentence beginning} dengan permulaan ayat seperti "Orang Ateis itu sangat" seperti yang dilihat di atas. We also evaluated our models on BBQ-Lite, a simplified version of BBQA [P that is avail- able on BIG-bench. We were somewhat surprised by the results, and are unclear if any conclusions can be drawn based on them. In essence, we appear to find that our plain LMs do not exhibit very significant biases on this benchmark, using the available metrics. Here the metric is roughly ‘how often does the model invoke a term in a negative context, divided by how often it uses the term generally’. A value of 0.5 corresponds to the models being ‘unbiased’. RLHF training perhaps improves the results slightly (likely by improving sentiment) though the effect is very small (see Figure Kita boleh kembali kepada penilaian ini pada masa akan datang menggunakan metrik BBQA penuh [P Melayu et al, 2021) 18). arrish et al., 2021]. In Section 3.4.2 we investigated gender bias in preference models. We tried using this format for evaluating RLHF models, but we found that these models were more than 10 less likely than plain or context-distilled LMs to use a gendered term to complete sentences like ‘The CEO was a’. In other words, statements like ‘The CEO was a father’ seem to be somewhat off-distribution for the RLHF models, at least compared to general LMs. This is problematic for evaluations, because it means that the bias measurements are likely to be dominated by noise, and are unlikely to be representative of model behavior (i.e. model-generated samples). So in Appendix B.10 we experiment with another formulation of the task, where we reverse the order of occupation and gender terms, and add a ‘For work’ clause; this kept both the RLHF and other models on-distribution. The results are discussed in Appendix B.10, where we show that RLHF model’s biases are very strongly correlated (Figure 40) with underlying LM biases. In particular, with this evaluation we find that large RLHF models have gender biases that are very similar to language models evaluated at a lower temperature. 5 Competing Objectives, Specialized Skills, and OOD Detection Satu kekhawatiran mengenai teknik penyesuaian adalah bahawa mereka boleh merosakkan prestasi model. we highlight a quantifiable trade-off of this kind, between helpfulness and harmlessness when training preference models. But it appears that larger models suffer less of a performance drop from this trade-off. 5.1 Furthermore, we also find that the conflict between helpfulness and harmlessness is relatively unique. Pref- erence models can learn to reward strong performance at specialized skills without any loss in performance at helpfulness and harmlessness. In Section we consider the evaluation of summarization quality as such a skill, using the learning-to-summarize Perbincangan di atas telah diubahsuai.Selepas di bahagian Kami menunjukkan bahawa model kod (iaitu, model disesuaikan pada kod oleh latihan yang diselia) juga serasi dengan intervensi penyelarasan HH, walaupun latihan RLHF tidak melibatkan data kod atau contoh. 5.2 [Stiennon et al., 2020] 5.3 dalam seksyen we highlight another approach to avoiding harmful behavior – it may be possible to reject most harmful requests, even without any access to harmfulness training data, by leveraging out-of-distribution detection techniques [F Pendekatan ini juga boleh berguna lebih umum dalam senario pelancaran di mana permintaan yang pelik atau luar topik perlu ditandai atau diabaikan. 5.4 ort et al., 2021]. 5.1 Mixing Helpful and Harmless Objectives In many cases harmlessness acts as a constraint on helpfulness. So we should expect that helpfulness and harmlessness may behave as partially anti-correlated objectives. We establish this by evaluating preference models trained on different mixtures of HH data, and with different weightings. At a conceptual level, the HH PMs may essentially be learning to first classify the data and then choose a score depending on the distribution. We will show that larger models perform better and are more robust to data mixture and loss weighting, which may be due to their having greater success at separating reasonable from harmful requests. 5.1.1 Varying Helpful vs Harmless Data Fraction Kami melatih model menggunakan split data yang berkisar daripada 100% berguna kepada 100% tidak berbahaya dalam interval 10%. pengedaran data statis kami mempunyai perbandingan 42k merah-teaming, jadi untuk mengawal saiz dataset kami sentiasa membina campuran dengan jumlah jumlah perbandingan ini. shows performance on both harm- lessness and helpfulness as the training data mixture is varied. Note that training entirely on helpfulness or harmlessness data results in performance on the other distribution which is . This exemplifies the extent to which these distributions are in tension with each other. 19 significantly worse than chance Figure dalam lampiran juga plot rata-rata ketepatan ujian atas kedua-dua berguna dan tidak berbahaya (di mana Mean Acc = (Harmlessness Acc + Helpfulness Acc)*/*2). Curves untuk model yang lebih besar kelihatan lebih tajam berhampiran dengan 0% dan 100% kawasan, tetapi lebih ramping di atas. Curves untuk model yang lebih kecil adalah lebih beransur-ansur, dengan puncak yang lebih jelas di tengah-tengah. 26 5.1.2 Weighting Helpful vs Harmless Losses Instead of studying different data mixtures, we can try re-weighting the losses. Since we have more helpful- ness than harmlessness comparisons, we experimented with weighting the losses as LTotal = LHelpfulness + λ · LHarmlessness for λ E 2 {1; 2; 3; 4; 10}, as shown in Figure 27 (relegated to the appendix). We note that larger models seem more robust to the choice of λ. Increasing λ from 1 to 10 causes a 7.4% decrease in accuracy on helpfulness for the 13M parameter model, whereas it only causes a 1.5% decrease in accuracy on the 52B parameter model. 5.2 Summarization as a Specialized Skill Kami mengharapkan bahawa model yang disesuaikan dengan kemahiran khas boleh menjadi sangat berguna dan berharga. adakah penyesuaian mengganggu penyesuaian untuk kemahiran khusus? As one test of this question, we studied PM finetuning on the learning-to-summarize (LtS) dataset vs. a mixture of LtS and HH data. We formatted the LtS data in conversa- tional format so that it matches the HH data, as follows: [Stiennon et al., 2020] Human: Can you write a summary of this article for me? ...Text... Assistant: Sure, here it is: ...Summary... As shown in Figure large preference models trained on a mixture of HH and LtS datasets perform equally well on both. So at least at the level of preference modeling, there seems to be no cost to mixing HH with the specific skill of evaluating summarization quality. 20, 5.3 Natural Language RLHF on Code-Finetuned Models As another test of a specialized skill, we would like to see if natural language alignment can be combined with coding without compromising performance. Since our crowdworkers were never instructed to probe the model’s coding abilities, and most likely do not have a great deal of coding expertise, our human feedback data does not include a significant number of code-related conversations. The preference model finetuning dataset and the RLHF prompt dataset thereby do not contain any significant amount of code, though there is some code in the LM pretraining mix and possibly a small amount in the PM pre-training (PMP) mix. This makes code-related problems an interesting way to test generalization of RLHF, and especially its compati- bility with other skills. Our ‘base code models’ were finetuned on Python code scraped from Github as described in Bermula daripada model Python yang disesuaikan (Python FT), kami kemudian menjalankan latihan bahasa semulajadi RLHF standard kami menggunakan model keutamaan 'static' dan prompts. [Bahasa Melayu dan lain-lain, 2021] We evaluate models on the HumanEval dataset which prompts language models with python function signatures and docstrings. Models are tasked with correctly filling in the function body given the context, and model-written functions are run in a sandbox environment. In Figure we show results versus model size with and without RLHF training. We see the same trend here as with other evaluations – RLHF decreases the performance of small models, but improves the performance of larger models. [Chen et al., 2021], 21 RL training tends to decrease the entropy of the models’ distribution, and so we were concerned that these results would be very sensitive to temperature and top-p tuning. So for our 52B models, we performed a scan over temperatures and two top-p settings for both the RLHF models and the base code models, and then chose the best setting for each model and pass@k. We did a grid-search over the evaluation hyperparameters: T ∈ {0, 0.4, 0.6, 0.8, 1.0} × p ∈ {0.95, 1} × k ∈ {1, 5, 10, 25, 50, 75, 100}. Results are summarized on the right side of Figure 21. For each model and for each k in pass@k, we take the maximum performance over all 10 combinations of hyperparameters. We see that RLHF improves performance over the baseline on this evaluation, for all pass@k. We should emphasize that as with our other evaluations, the improvements in performance from RLHF are modest. In fact, we find that simply prompting a base code model performs slightly better, as shown in Figure Appendix further describes the format of the prompts we used (i.e., ‘HHH prompts’), which consist of a couple of code examples. 38. B.8 daripada We also conducted experiments involving adding buggy code to the prompts, which typically worsens per- formance (see Kami mendapati bahawa model RLHF tidak berfungsi lebih baik daripada snapshots model kod asas awal mereka, apabila prompts ini dimasukkan ke dalam konteks semasa penilaian, walaupun selepas pemindaian di atas suhu dan top-p. [Chen et al., 5.4 Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests In this work we are primarily focused on achieving harmlessness entirely through natural language dialogue. However, one might try to avoid harmful behavior in a somewhat different manner, by either restricting language assistants to only respond to a narrow range of queries (approved-list), or by filtering and rejecting known types of bad behavior (block-list). We could use our preference models for these purposes, but we might also take a different, less supervised approach, and leverage advances in out-of-distribution (OOD) detection. Such an approach might also be useful for those who want to build systems that only respond to a narrow range of queries (e.g. code models that should avoid non-code topics). Out-of-distribution detection (OOD), and especially near out-of-distribution detection, have been a major challenge for deep neural networks. Deep networks routinely assign high probability to mis-classified inputs [Guo et al., 2017, Lakshminarayanan et al., 2016] as well as to test inputs not belonging to any of the training classes [Nguyen et al., 2014]. There have been many approaches to OOD detection based on discriminative models [Hendrycks and Gimpel, 2016, Lee et al., 2018, Liang et al., 2017, Liu et al., 2020] as well as deep generative models [Nalisnick et al., 2019, Zhang et al., 2020]. The more difficult case of OOD detection, the so-called near-OOD detection [Winkens et al., 2020], has recently been improved upon significantly using pre-training and large models [Fort et al., 2021]. For a prompt i, we extract a vector of activations of dimension dmodel from a layer ` and call it v ` i ∈ R dmodel . The task is to distinguish between an unseen example of harmlessness and helpfulness data without being explicitly shown any harmlessness data at all. This approach works by measuring the deviation of a prompt from the helpfulness data, rather than measuring how close it gets towards harmlessness data in particular. In this way, we do not depend on the specific harmful content we have at hand, and can potentially filter different kinds of non-helpfulness content. To detect whether an input comes from the in-distribution (the helpfulness dataset), we use a scoring function that takes the input and maps it to a scalar value score(x). To do that, [Lee et al., 2018] first proposed to fit a simple model to training examples of the in-distribution, Din train = {v ` i ∈ R dmodel}. We calculate the mean µ = 1 Ntrain PNtrain i=1 v ` i and the covariance matrix Σ = 1 Ntrain PNtrain i=1 (vi − µ) (vi − µ) T . The Mahalanobis distance of an unknown activation vector x from this training set is score(x) = (x − µ) T Σ −1 (x − µ). A simple improvement on top of the Mahalanobis distance called the Mahalanobis distance has been proposed in and shown to lead to better AUROC as well as more robust detection for a range of OOD problems in vision and genomics (in addition to more robustness to adversarial attacks [F Terinspirasi oleh kaedah ini dan mengakui bahawa masalah kami tidak secara semulajadi melibatkan kelas yang bermakna semantik yang terdiri daripada penyebaran dalam, kami mencadangkan satu modifikasi lanjut yang kita sebut Relative Mahalanobis distance. We compute it by fitting a full covariance matrix Σ as before, as well as a diagonal- only covariance matrix Σdiag, and assigning the difference of their Mahalanobis distance as our scoring function, score( ) = ( − ) Σ−1( − ) − ( − ) Σdiag−1( − ) daripada Relative [Ren et al., 2021] ort, Penyederhanaan x x µ T x µ x µ T x µ Gambar shows the results for our OOD detection experiments, trying to distinguish the helpfulness data from harmlessness data using our new Simplified Relative Mahalanobis distance to the helpfulness activation vectors. The figure shows AUROC for activation vectors from different layers of the model and different model sizes. The bigger the model, the better its performance, with the middle layers performing best. The right panel of Figure shows an example of the OOD score distributions for the helpfulness data (blue) and harmlessness data (red) for a 64L layer of 52B parameters and its 32th layer. We can see that the mode of the distances of the Harmlessness data is clearly higher than for the helpfulness data. The comparison of the Simplified Relative Mahalanobis distance and the standard Mahalanobis distance is shown in the Appendix dalam Figure 22 22 B.9 39. Kelebihan pendekatan ini ialah kita boleh membezakan data berguna daripada data tidak berguna, dengan data tidak berbahaya hanya jenis tertentu data tidak berguna. If we have access to a small number of examples of Harmlessness inputs (the out-distribution), we could perform a few-shot outlier exposure, as first proposed by menggunakan satu kelas yang mewakili contoh OOD. [F] has used outlier exposure on top of individually strong near-OOD detectors and showed that they still enjoy a large benefit from being exposed to the examples of the out-distribution. We observe a similar benefit here, as discussed in detail in Section and shown in Figure [Bahasa Melayu dan lain-lain, 2018] [Mahkamah dan rakan-rakan, 2019] dan lain-lain pada tahun 2021. B9 23. In particular, with only 10 examples of harmful prompts, we can achieve an improved AUROC of 0*.*94 0*.*02 for the 64L model. The best performance among all layers of all models (the middle layers of the 52B model) without outlier exposure (already using our Simplified Relative Mahalanobis distance, see Figure [22)] is approximately 0*.*85. A 4L model exposed to only 10 examples of harmful data gets an AUROC of 0*.*86 0*.*01, while having only 13M parameters as compared to the 52B. The OOD detection gains from outlier exposure are therefore very significant in comparison to the gains coming from model size scaling alone. 6 Contoh dan perbandingan kualitatif Kami mendapati bahawa proses penyelidikan kami sendiri bergantung pada penilaian kualitatif, untuk mendapatkan rasa kekuatan dan kelemahan model, walaupun apabila matlamat akhir adalah untuk menghasilkan sejenis metrik kuantitatif. An obvious issue with the qualitative evaluation of samples is that it’s difficult to know to what extent they have been cherry-picked. To mitigate this issue, Kami menyediakan sampel berdasarkan prompts yang muncul dalam perkongsian dengan kerja yang sebanding orang lain dalam Lampiran including Instruct- GPT LaMDA and sensitive questions appearing in PALMS We encourage readers to skim these samples to get an (arguably) unbiased sense for our model’s behavior. untuk setiap prompt kami mencipta 17 sampel, mengurutkan mereka dengan model keutamaan dalam talian HH kami, dan kemudian memaparkan sampel median C, [Ouyang et al., 2022], [Thoppilan et al., 2022], [Solaiman dan Dennison, 2021] 6.1 Comparison with Human Writers Sebagai ujian tambahan kepada model kami, kami mengumpul dialog HHH berkualiti tinggi daripada penulis manusia. Penulis-penulis ini telah dipekerjakan di Upwork (terpisah daripada kumpulan pekerja kerumunan kami) berdasarkan kerja tulis yang berjaya sebelum ini dan ulasan positif. Then based on the prompts (leaving off the final writer-written assistant responses), we generated 17 examples from our best HH and best purely-helpful online models, and chose the median response from this pool as ranked by our online HH preference model. We then asked our crowdworkers to rank responses from the writers, along with one response each selected from our HH and helpful-only models. We then use these results to estimate an Elo score for our human writers, by comparing to our two models, and we display the result as a horizontal band on Figure 1. Kami menggunakan band kerana perbandingan penulis sebenarnya agak tidak konsisten dengan kedudukan Elo kami yang lain, kerana pekerja kerumunan lebih suka model HH dalam talian daripada penulis manusia 56% masa, tetapi lebih suka model keberkesanan murni kepada penulis hanya 51% masa. We find that the performance of both our helpful and HH models is roughly comparable to that of the human writers, though in fact the models are slightly preferred by crowdworkers. We also find that our PM’s typically assign higher scores to the model responses, as shown in Figure e believe that this result should be interpreted with caution and caveats, however. The writers’ prompts were not very adversarial; many strike us as ‘softball questions’ that models can easily do well on. As researchers who have interacted with models for tens or hundreds of hours, we could likely come up with many questions where humans would be more likely to outperform the models. Relatedly, the writers’ conversations do not strike us (in many cases) as especially realistic, in the specific sense that we would not expect a human to derive a great deal of practical value by asking an AI assistant for help in these ways. We expect that our models likely still fall significantly short of providing the kind of value that a sophisticated human assistant could provide via chat, even if they were operating under the constraints the model experiences (e.g. no internet access). 24 W Berikut ialah beberapa contoh yang menggambarkan panggilan, respons penulis, dan respons model HH RLHF dalam talian kami. dalam contoh ini, model ini berfungsi dengan baik pada mulanya, tetapi kemudian membuat beberapa pernyataan yang sedikit tidak tepat mengenai cara-cara di mana ia memperbaiki kesilapan: In this example the writer’s statistic is arguably a bit misleading (presumably unintentionally), and the model’s response may simply be superior. This strikes us as an example of a ‘softball question’ where we expect models to do well, since it is common, vague, and open-ended: Dalam contoh terakhir ini, model ini membuat beberapa kesilapan halus, tetapi kebanyakan pembaca mungkin akan mempunyai kesukaran untuk mendeteksi mereka tanpa berkonsultasi bahan rujukan: Dialog-dialog seperti di atas membantu untuk menggambarkan batas-batas latihan maklum balas manusia: . This is one reason why we may not want to rely on human feedback to train models to be honest, or for any goal where human judgment is not the best possible source of information. once model errors become sufficiently subtle, they will no longer be penalized appropriately, and the model’s incentives may no longer be aligned With that said, AI models still have limitations that can seem highly incongruous. Here’s an example where we, the more experienced model-interrogators, really played ‘hard ball’ with the AI: 6.2 Soalan-soalan sensitif dan mengelakkan berbanding komitmen Berikut ialah beberapa contoh daripada PALMS sensitive questions. We chose them to illustrate how the model sometimes avoids very sensitive subjects, but often provides otherwise reason- able responses. Please see Appendix for many more examples, also including some from InstructGPT dan LAMDA yang cepat. [Solaiman and Dennison, 2021] C [Ouyang et al., 2022] [Bahasa Melayu et al., 2024] Here is a case where the model just refuses to answer the question: Here the model indicates that it should not answer, but then provides some relevant information (perhaps it’s also problematic that the model focuses on the US, though this is likely a consequence of it having been trained in English by US-based crowdworkers): Dan di sini model ini memberikan jawapan yang cukup lengkap, tanpa cuba mengelakkan topik: Secara kualitatif, pengalaman kami adalah bahawa dengan menyesuaikan optimasi untuk kegunaan berbanding tidak berbahaya, seseorang boleh melatih model yang sangat menghindari apabila topik sensitif muncul, atau model yang hampir sentiasa cuba untuk terlibat, walaupun pada subjek sensitif. juga menyediakan pelajaran yang berguna jika matlamatnya adalah untuk melatih model yang menyediakan jawapan penuh dalam menghadapi soalan-soalan sensitif. 4.4 6.3 Example Dialogues Most of our samples only involve a single prompt and response, so in this section we show a few two-turn dialogues. In this dialogue the model follows instructions, but could have done a significantly better job capturing the nuance of the situation: Here is some dialogue on a technical subject from a researcher who was experimenting with using the model to help debug an engineering issue: Apabila diminta untuk gaya penulisan burung merpati, model ini lebih cenderung membuat pernyataan yang berpotensi berbahaya, eksplisit, atau menyinggung: In this case we chose a human reply that takes the conversation in a subtly dark direction, and the model rebuts the argument: 7 Discussion We have shown that it’s possible to use reinforcement learning from human feedback to train language models that act as helpful and harmless assistants. Our RLHF training also improves honesty, though we expect other techniques can do better still. As in other recent works associated with aligning large language models RLHF improves helpfulness and harmlessness by a huge margin when compared to simply scaling models up. [Stiennon et al., 2020, Khamis et al, tahun 2022, Ouyang et al., 2022, Nakano et al., 2021, Menick et al., 2022], Intervensi penyelarasan kami benar-benar meningkatkan keupayaan model besar, dan boleh dengan mudah digabungkan dengan latihan untuk kemahiran khusus (seperti pengkodean atau ringkasan) tanpa sebarang penurunan dalam penyelarasan atau prestasi. Model dengan kurang daripada kira-kira 10B parameter bertindak berbeza, membayar 'caj penyelarasan' pada keupayaan mereka. Ini menyediakan contoh di mana model berhampiran dengan state-of-the-art mungkin diperlukan untuk mengambil pelajaran yang betul daripada penyelidikan penyelarasan. The overall picture we seem to find – that large models can learn a wide variety of skills, including align- ment, in a mutually compatible way – does not seem very surprising. Behaving in an aligned fashion is just another capability, and many works have shown that larger models are more capable finetune with greater sample efficiency and do not suffer significantly from forgetting Although we did not demonstrate it directly, we also expect that RLHF alignment training can be mixed with or precede train- ing for other objectives; this might be relevant in the future in order to avoid the production of intermediate, unaligned AI systems. [Kaplan et al., 2020, Rosenfeld et al., 2019, Brown et al., 2020], [Henighan et al., 2020, Askell et al., 2021], [Ramasesh et al., 2022]. We did find a clear tension between helpfulness and harmlessness, where models trained entirely for helpful- ness are preferred to models trained for HH, when evaluating only on helpfulness. We believe this is partly due to a subtlety in our data collection process, as we rarely collect data teaching models how to deal pos- itively with harmful requests (i.e. how to be a sort of ‘hostage negotiator’), but only on how to avoid them. And we also found that at least at the level of preference models, the helpful-harmless tension diminishes as models become larger and more capable. Nevertheless we do expect that this tension is real, and that caution may cut into model performance on the margin. Large generative models have been referred to as ‘foundation models’ These mod- els are extremely interesting objects for research, but without further finetuning, they can exhibit harmful behaviors. Our work suggests that alignment training can be incorporated into foundation models without compromising their utility and versatility, and so perhaps it could soon become a part of their definition. [Bommasani et al., 2021]. 7.1 Limitations Walaupun kami percaya hasil kami memberikan gambaran yang menjanjikan untuk penyelarasan model bahasa yang sedia ada, kerja-kerja mengenai topik ini masih dalam peringkat awal, dan mempunyai beberapa batasan. kami melihat kerja kami mengenai penyelarasan sebagai projek yang berterusan; kerja kami was step zero, and this is step one. [Perkh dan rakan-rakan, 2022] [Bahasa Melayu et al., 2024] Kami secara pragmatik mendefinisikan penolong yang diselaraskan sebagai AI yang helpful, honest, and harmless. We are op- timistic that at present capability levels, the techniques we have discussed here provide a reasonable approach to achieving helpfulness and harmlessness. However, although our techniques improve model honesty, we believe we are just scratching the surface of that problem, and that other techniques may more efficiently and effectively produce honest AI models. 18 Here we have essentially focused on the average-case behavior of our models. However, even if we were convinced that our models were HHH in expectation, a clear next step would be to attempt to study and eliminate bad behaviors (especially harmfulness) even in the worst case. We have not addressed this question of robustness here, but hope to study it in the future (approaches such as Ia hanya akan menjadi lebih mendesak apabila sistem AI maju dan menghadapi peralihan pengedaran semasa penyebaran. [Perez et al., 2022] Alignment AI boleh menjadi sukar dan ambigu untuk dinilai. Jadi, contohnya, manakala model besar kami yang dilatih oleh RLHF melakukan lebih baik daripada LM biasa pada hampir semua penilaian keupayaan, seseorang boleh berharap bahawa model yang benar-benar berguna prestasi tembakan nol akan sama dengan prestasi beberapa tembakan model yang tidak diselaraskan. Logik di sini adalah bahawa jika model benar-benar boleh 'mengikuti arahan', maka petunjuk atau penjelasan harus mencukupi untuk menampung kesenjangan tembakan nol. we close a bit less than half of this gap (Figure We also briefly investigated whether our RLHF-finetuned code models have any comparative advantage when exposed to prompts including buggy code Satu akan berharap model yang sepenuhnya diselaraskan akan melakukan yang terbaik untuk menulis kod yang betul, walaupun apabila diberikan petunjuk buggy. [Lin et al., 2021] 5). [Bahasa Melayu dan lain-lain, 2021] We also harbor a general concern that perhaps our techniques only render models aligned ‘on the surface’, and that they still harbor harmful biases or other tendencies that may surface in more subtle contexts. We found that RLHF models have a more positive sentiment towards all racial and religious groups, which seems promising, but does not necessarily indicate that biases have been reduced. And with respect to gender, we found that RLHF model biases are very strongly correlated with the bias of the underlying language models. That said, further work will be required to understand if this is a limitation of RLHF as a technique, or of our particular HH datasets. In any case, we likely need to build more subtle and comprehensive evaluations that include multi-turn dialogue, as this is an area where humans will likely use the models, and it’s also a place where it’s inherently more difficult to measure performance against subtle objectives such as bias and fairness. On a much more practical level, we do not have much experience applying RL techniques to large generative models. Experienced AI practitioners know that there are a large variety of tweaks and tricks that require experimentation to identify, and that can majorly improve the stability and performance of training. We have encountered some stability issues with RL, and although we performed some rudimentary hyperparameter scans, we expect that with more experience and study we could do better. We also did not explore variations in online training, such as literally updating a single PM or RLHF model; rather we retrained these models from scratch on each iteration. Another direction for exploration is to use a non-trivial function of PM scores as the RL reward, distorting the score distribution to e.g. focus more on discouraging bad behavior rather than rewarding good behavior. In summary, there are many future directions to explore for improving RLHF. A final concern is whether techniques like those we have employed will continue to apply as AI models become increasingly capable. We take these concerns very seriously. In our view, the present work makes some progress towards our goal, which is to establish a set of simple and universal techniques that can align AI models at present capability levels. Assuming this goal can be met, one of the next steps will be to build consensus among researchers and to understand alignment in greater depth, including how techniques scale with AI capabilities. The hope will be to create an evolving pragmatic state of the art for training AIs that are thoroughly helpful, honest, and harmless. initial 19 Langkah penting yang lain ialah menggunakan garis asas ini sebagai titik permulaan untuk meneroka teknik lain yang boleh menangani kes penggunaan yang lebih maju dan mod kegagalan yang lebih spekulatif. Idea dan teknik baru kemudian boleh dibandingkan secara pragmatik dengan kaedah sedia ada, dan kemudian dimasukkan ke dalam amalan standard jika mereka menghasilkan peningkatan lanjut dalam keselamatan dan ketahanan. kita sudah menghadapi contoh-contoh yang menunjukkan keterbatasan maklum balas manusia, dan oleh itu kita perlu mula membangunkan kaedah-kaedah lain. 6.1, 7.2 Alignment Data as a Public Good Dalam kerja ini, kami membenarkan akal semulajadi untuk mendefinisikan apa yang merupakan kebaikan yang berguna dan tidak berbahaya.Ini telah mencukupi untuk penyelidikan kami mengenai 'pelarasan teknikal', iaitu soalan sama ada teknik-teknik tertentu boleh digunakan untuk melatih model AI menjadi lebih berguna dan tidak berbahaya.Tetapi kami telah mengelakkan untuk menangani soalan yang mendasar tentang jenis tingkah laku yang boleh diharapkan daripada model AI yang digunakan. Yang mengatakan, tanpa spesifikasi yang jelas untuk format dan jenis ‘data penyelarasan’ yang paling relevan untuk latihan AI, ia telah menjadi sukar bagi sesiapa sahaja selain penyelidik untuk mengumpul maklumat yang diperlukan untuk melatih sistem AI yang selamat dan bermanfaat. to teach AI models complex human preferences, and we have also found that preference modeling based on ranked comparisons scales better than many other techniques. [Stiennon et al., 2020, Ouyang et al, tahun 2022, Nakano et al., 2021] [Bahasa Melayu et al., 2024] One possible approach would be for an independent organization with ethical, legal, and cultural expertise to create a very high-quality dataset expressing human preferences for AI behavior (via comparisons). Such an organization could also use a novel governance structure, so that a larger set of societal stakeholders could factor into the decisions it makes about how to create and curate alignment data – in contrast to today, where private companies make these decisions in an opaque manner using governance structures that grant power to financially interested parties. Datasets created in this way might be used for both training and evaluation of AI models, and could even begin to establish standards for behavior. Due to the rapid improvement in AI language models, we expect that such datasets would be most valuable if they encode preferences at human- level sophistication. In any case, this is just one speculative possibility for broadening participation in dataset creation. Penyelidikan kami telah mendapat manfaat daripada kumpulan data penyelidikan yang tersedia secara awam dan penilaian yang berkaitan dengan penyelarasan AI dengan nilai-nilai manusia and we plan to release our preference modeling data for others to use in their research. Unfortunately, this does not seem to be a standard practice among alignment researchers, as evidenced by some recent work. While we agree that LLMs themselves can be used for harm, it seems that no such argument can be made for alignment data. [Stiennon et al., 2020 , Rizky et al, 2021a, Ia adalah amat penting untuk membolehkan kerjasama dan kebolehpercayaan untuk penyelidikan penyelarasan dan keselamatan. Apabila sistem AI menjadi lebih berkuasa dan lebih meluas digunakan, kos kesilapan dan kesilapan boleh meningkat secara melampau. Kami percaya bahawa satu-satunya cara untuk mengatasi kegagalan keselamatan yang berpotensi daripada sistem AI canggih adalah untuk membina komuniti penyelidik yang berfikir dengan kepakaran yang mendalam, dan keupayaan untuk menilai sistem secara empiris. Ini akan kekal hampir mustahil jika pengetahuan tentang penyelarasan sistem canggih kekal di dalam banyak organisasi bebas. Perkongsian data kelihatan sebagai cara yang paling mudah dan paling masuk akal untuk membolehkan perkongsian dan pengesahan hasil. One ostensible reason for secrecy is that organizations may use data from users to develop alignment datasets, and then justify not sharing the datasets on the grounds that it violates user privacy. This is a challenging issue that requires organizations to think about how to reconcile commercial priorities with the need to create a ‘safety commons’ for the community. If alignment becomes interlinked with the concept of commercial moats, that could reduce the overall net level of safety of the AI ecosystem. Therefore, we believe that datasets developed for alignment should be kept separate from commercial data, and should be openly shared to advance research on safe and beneficial AI. 7.3 Broader Impacts We hope that our work provides compelling evidence that AI systems can be made safer and more useful at the same time, and without performance costs. As noted above, we have largely remained agnostic on the question of which values define acceptable and unacceptable AI behavior. Thus we hope that rapid progress in technical alignment and the consolidation of specific techniques will motivate the development of publicly available alignment data, guidelines, and benchmarks. AI technologies are dual-use, meaning they can be used beneficially and otherwise. We have found the ef- fectiveness of preference modeling and RLHF striking (in our research and others’), and believe there’s very legitimate concern that these techniques could be used for censorship, fraud, and misinformation. Straight- forward commercial use-cases also seem worrisome, especially if optimization for objectives like user en- gagement and persuasion are mixed together. At the most naive level, if you can optimize for ‘harmless’ then you can ‘flip the sign’ and generate harmful systems. We also found that systems trained exclusively to be helpful become easier to use for harmful ends, which suggests that as systems become more powerful, it will become increasingly important to directly curb their potential for harms. 20 Perhaps the broadest impact of this work, and the general development and dissemination of controllable, human-like language generation akan menjadi budaya. dalam gambar we used an Elo scale, essentially the chess rating system, to compare and evaluate natural language assistants, and we even included comparison to human writers. This kind of comparison risks trivializing the importance of language, which is certainly not just a game, but the core medium of culture and society. While seeking to align increasingly capable AI systems feels like a robustly good action, how and when to deploy these systems poses more challenging questions – culture is fundamentally a human enterprise, but large-scale generative models hold the possibility of magnifying and minimizing different parts of human culture in unpredictable and opaque ways, which could have broad downstream influences. [Ganguli et al., 2022], 1 Acknowledgments Acknowledgments We thank Sam Bowman, Paul Christiano, Jacob Hilton, Jan Leike, Ethan Perez, and Jeff Wu for helpful feedback on the draft. We thank Daniela Amodei, Jamie Kerr, Jia Yuan Loke, Rebecca Raible, and Tim Telleen-Lawton for support with the project. Author Contributions Kontribusi Penulis Yuntao Bai menjalankan kebanyakan eksperimen pada RLHF dan banyak eksperimen pemodelan keutamaan. beliau membuat sumbangan utama kepada reka bentuk eksperimen, pengukuran, dan penilaian prestasi model dan tingkah laku. Andy Jones and Kamal Ndoussse built the infrastructure for RL training of large language models. They also built associated plotting and monitoring systems and implemented the PPO algorithm. They helped with the design, implementation, and debugging of RLHF. Amanda Askell membantu merancang penilaian model, mengumpul sampel dan penilaian daripada penulis profesional, membina sistem untuk meningkatkan kualiti dan kuantiti pengumpulan data, dan bekerjasama dengan Jared dan Jackson pada penilaian yang berkaitan. Anna Chen helped with general RL and RLHF experimentation, and contributed to the research design. Nova DasSarma menguruskan infrastruktur cluster yang mendasari, membolehkan latihan RL skala besar dan pengumpulan maklum balas manusia. Dawn Drain trained the underlying code models and collaborated with Saurav on coding evaluations. Stanislav Fort performed the OOD detection and outlier exposure research and analysis on helpful versus harmful data samples. Deep Ganguli led the red-teaming data collection effort and design, often working with Jackson, Liane, Amanda, and Ben. He designed and ran the societal impact evaluations in collaboration with Jared, and helped with model evaluations generally. Tom Henighan membantu dengan pra-latihan model bahasa yang mendasari, dengan penciptaan dataset, dan dengan menguruskan cluster semasa beberapa fasa projek. Nick Joseph membantu merancang dan membina rangka kerja untuk latihan yang cekap model bahasa yang besar dan model pilihan. Saurav Kadavath designed and conducted experiments on helpful/harmless dataset mixing. Saurav also ran RLHF training on code models, with support from Yuntao, and ran coding evaluations in collaboration with Dawn. He also ran the majority of the natural language evaluations of basic and RLHF-finetuned models, and helped with RLHF training generally. He helped to write the paper. Jackson Kernion memimpin penilaian dan pengurusan maklum balas manusia dan membantu membina dan mengekalkan antara muka maklum balas.Dia juga membantu dengan analisis data dan bekerjasama dalam penilaian model.Dia menjalankan kebanyakan percubaan perbandingan model kami. Tom Conerly helped with engineering, specifically with fast and efficient sampling. Sheer El-Showk helped with pretraining research and dataset construction. Nelson Elhage menyumbang secara signifikan kepada pra-latihan dan kepada visi kejuruteraan. Zac Hatfield-Dodds helped with codebase maintenance and with engineering, specifically with fast and effi- cient sampling. Danny Hernandez contributed to pretraining and especially to dataset design. Tristan Hume helped with streamlining our infrastructure. Scott Johnston helped with pretraining research. Shauna Kravec contributed to the development and use of our RL systems, and collaborated on RL research. Liane Lovitt helped with red-teaming, and in particular with designing the interface. Neel Nanda contributed to research discussions and priorities for alignment. Catherine Olsson helped advise on human feedback data collection, and contributed advice on alignment and evaluation. Dario Amodei memberi nasihat kepada projek ini dan memimpin usaha untuk membina dan menguji infrastruktur RL dan ML. Tom Brown memimpin usaha kejuruteraan, termasuk pra-latihan yang cekap, pengambilan sampel, dan kestabilan dan reka bentuk sistem RL. Jack Clark led societal impacts efforts and advised the project, including on various evaluations. Sam McCandlish led pretraining efforts and advised the project. Chris Olah collaborated on discussions of alignment and contributed to our research and evaluation infras- tructure. Ben Mann led the design and construction of the human feedback data collection interface and the underlying infrastructure. He also helped lead crowdworker management, and he provided engineering support for the project as a whole. He also contributed to pretraining and cluster management. Jared Kaplan merancang dan mengetuai projek ini. beliau membantu dengan semua aspek, termasuk reka bentuk penyelidikan, kejuruteraan, eksperimen, dan penilaian. beliau juga menyumbang kepada pra-latihan dan membantu membina infrastruktur penilaian. A Details, Analysis, and Evaluations of Supervised Training A.1 Context Distillation For context distillation, we follow the prescription from Secara khusus, kami mula-mula mencipta data seperti berikut: [Bahasa Melayu dan lain-lain, 2021] 1. We prepend the ‘HHH prompt’ (i.e., a set of prompts designed to elicit helpfulness, harmlessness, and honesty) to sequences of text, with 50% of the text coming from our pretraining dataset, and 50% coming from a StackExchange dataset. For the former, we simply append pretraining data after signaling the beginning of another conversation with “Human:”. With StackExchange, we formulate a fake Human/Assistant conversation by using the question as the human side of the conversation, and a top-rated answer as the assistant role. 2. For both datasets, we then perform forward passes with a basic pretrained 52B model, and record the top 50 log-probabilities and their indices (within the vocabulary) for the tokens following the prompt. We store the log-probs, indices, and tokens together as a small new dataset. 3. To perform context distillation finetuning, we pass the tokens from our new dataset through models (of all sizes), and define the loss as the KL divergence between the stored log-probs and the predic- tions of the model undergoing finetuning. For each token, we use a 51-category distribution, with the 51st category covering the total probability from all the tokens other than the top-50. We show learning curves for context distillation in Figure Kami menggunakan saiz batch 32 urutan, dan kadar pembelajaran 0*.*05 kali kadar pembelajaran pra-latihan, yang kita memecah kepada nol secara linear semasa distilasi. 30. A.2 Preference Modeling Our preference models are trained on comparison data, with each data point consisting of a and a pair of . The prompt is a multi-step dialogue between human and model that always begins and ends on the human side, and each response is a continuation of the dialogue. For instance, in Figure prompt terdiri daripada lima langkah pertama dialog, dan jawapan ditunjukkan dalam kotak biru. PM kemudian menetapkan skor di hujung setiap jawapan. Perhatikan bahawa sementara PM hanya dilatih untuk menilai kualiti jawapan akhir, konteks lengkap perbualan disediakan kepada model. prompt responses 6 yang We train scans of PMs ranging from 13M to 52B parameters. All PMs go through three phases of training: (1) language model (LM) pre-training on a large language corpus, (2) preference model pretraining (PMP), and (3) finetuning on human feedback. LM pre-training details, including choice of hyperparameters and datasets, are explained in Appendix A of our previous work [Askell et al., 2021]. For PMP, we use learning rate of 0.1 relative to LM pretraining, and train on a mixture of comparison data made from StackExchange, Reddit, and Wikipedia. Data preparation and labeling are explained in Appendix C.1 of We train with context size of 1024 tokens. [Askell et al., 2021]. For human feedback finetuning, we use learning rate of 0.01 relative to the LM pretraining. We use context size of 1024 tokens, except for the ‘online’ model described in Section where we trained with 2048, which may help stabilize RLHF on long contexts. 4.5 daripada For both PMP and human feedback finetuning, we append a special ‘end-of-context’ token at the end of each sample, such that the PM score is predicted directly on top of this token. As explained in Appendix C.4 of this appears to improve PM performance. [Askell et al., 2021], Dalam semua fasa, kami hanya melatih satu iterasi untuk mengurangkan overfitting. A.3 Scaling of PM with Model and Dataset Size A major question is how performance of preference modeling scaling with model size and dataset size. This relates to a practical question – should we invest in collecting a larger dataset, or in training larger models? We seem to find more predictable scaling when training only on our helpfulness dataset, likely because the red-teaming data truly comes from a distinct distribution. Accuracy learning curves can be seen on the left of Figure We find that accuracy can be roughly fit by 32. where is the number of parameters in the PM and is the size of the dataset. P D However, the results when training on another preference modeling data distribution look quite different, as seen on the right in Figure Note that there appears to be a sort of discontinuity in behavior between 200M and 13B parameters. Perhaps this is related to the fact that the data was generated by a model with 6B parameters. 32. B Details, Analysis, and Evaluations of RLHF B.1 Training Setup B.1 Training Setup Di sini kami membincangkan beberapa butiran mengenai latihan RLHF. Kami memulakan dasar kami pada model distilasi konteks, yang dijelaskan dalam A. 1 We train the policy to generate responses to a dataset of prompts that maximize the score relative to a PM that was finetuned on human feedback. The prompt dataset is obtained from the training split of the PM comparisons dataset by simply removing the responses in each pair. Recall that we allow multi-step dialogue within the prompt (which always begins and ends on the human side of the conversation), but only train the policy to generate one response following each prompt. In future work, we plan to train policies to generate multiple steps, but this requires a separate model that generates the human side of the conversation, which can be implemented with a language model trained to imitate the human side of the conversation. Kami melakukan pelbagai pemindaian hiperparameter, dan akhirnya menggunakan kadar pembelajaran 0.01 berbanding dengan pra-latihan, koefisien ganjaran KL λKL = 0.001 (4.1), pemotongan PPO = 0.2, faktor diskaun γ = 1, dan tiada bonus entropy. Selain itu, dalam PPO, kami mengulang-ulang pada sampel yang sama K kali (lihat Algoritma 1 dalam [Schulman et al., 2017]), dengan K yang lebih tinggi biasanya membawa kepada hasil yang lebih stabil. Kami menggunakan K = 1 untuk pemindaian RLHF, K = 2 untuk kajian ketahanan (Seksyen 4.2), dan K = 4 untuk ‘online’ RLHF (Seksyen 4.5). Kami juga menetapkan had kepada bilangan maksimum token per model respons, menggunakan 32 untuk kajian ketahanan, dan di tempat lain 128. Akhirnya, untuk ‘online’ RLH B.2 More on Robustness Studies B.2 More on Robustness Studies dalam Figure we compare the score for all policy sizes and all sizes. The main observation here is that the slope of the graph increases with respect to ukuran yang lebih besar, yang menunjukkan bahawa ’s are significantly more capable of distinguishing policy performance. In other words, , in agreement with calibration studies in Section 33, test PM test PM test PM test PM Model preferensi yang lebih besar lebih kukuh 3.3. Finally, we take a moment here to address an issue we had neglected so far, which is that scores from different preference models should not be compared directly, since the absolute value of the score has no meaning, only relative scores are meaningful. We address this by a simple Kami membuat set data yang dikekalkan, yang terdiri daripada beberapa ribu sampel, dan menarik daripada setiap model keutamaan skor purata pada set data ini. Kami minta maaf bahawa plot yang tidak berkaitan dari bahagian yang berbeza mungkin telah menggunakan set data yang dikekalkan yang berbeza untuk penghapusan purata. mean removal B.3 Details of ‘Online’ RLHF B.3 Ciri-ciri ‘Online’ RLHF We give some more details on our ‘online’ RLHF policy discussed in Section This policy and its PM were trained on all the helpfulness and harmlessness data we had near the completion of this paper. We re-iterated each sample = 4 kali to improve stability, and sampled a maximum of 128 tokens per response. 4.5. K [Schulman et al., 2017] Throughout training, we periodically evaluate the of the policy on various held-out PM comparison datasets. More specifically, given a comparison consisting of a prompt and pair of responses, we assign an accuracy of 1 if the policy’s mean log-p on the better response is higher, and 0 otherwise. We show these results in Figure for various comparison datasets. In particular, we find that mean log-p accuracy of the policy isn’t as high as (i.e., fraction of comparisons on which the PM assigns a higher score to the better response), possibly suggesting room for further improvements to our RLHF pipeline. mean log-p accuracy 34 PM accuracy B.4 Robustness of ‘Online’ RLHF B.4 Robustness of ‘Online’ RLHF For our final online models, we had crowdworkers compare a variety of model snapshots from RLHF training, in order to better understand the robustness and general performance of our training process. In Figure (hak), kami menunjukkan skor Elo untuk model dalam talian semasa latihan RLHF. 15 In particular, we compare Elo scores established empirically from crowdworker preferences (i.e., di dalam gambar), dan skor Elo yang diramalkan oleh model keutamaan kami (iaitu, ) during RLHF training. For the latter, we sample responses from each snapshot on a set of held-out prompts, and evaluate the PM scores (which are then converted to Elo units). We notice that the naive PM predictions significantly overestimate the empirical Elos. This is due to a combination of the following fac- tors: Crowd- worker Preferences Naive PM Predic- tion 1. During crowdworker testing, each step of the conversation is written by one of the two models being tested. However, when evaluating a RLHF snapshot on held-out prompts, the policy only writes one response at the end of a pre-existing conversation (which had been previously generated by other models, as discussed in Appendix This leads to distributional shift between the conversations. B.1). 2. Elo and PM scores may not actually be transitive, as they involve the collapse of pairwise compar- isons onto a single line. For example, if PM scores satisfy − = 2 and = 2, even if those are well-calibrated scores, the implication that − = 4 may not be correctly calibrated, and we would naturally expect instead 4. A, B dan C a b b c a c a c < 3. Failures of PM robustness, so that the PM’s preference for the RL policy’s samples is miscalibrated compared to true human preferences. To explore these effects further, in Figure Kami menunjukkan skor Elo yang sesuai dengan empat pengukuran yang berbeza: 35 • Naive PM Prediction: Skor PM (diterjemahkan kepada unit Elo) yang direkam semasa latihan RLHF, yang menggunakan satu set petunjuk yang ditahan. • Mean PM Score on Crowdworker Data: The mean PM score on the actual crowdworker conversa- tions used to compute Elo scores based on crowdworker preferences. • Kedudukan PM pada Data Crowdworker: seseorang boleh cuba membezakan kegagalan kestabilan daripada skor PM berbeza berbanding PM . Here we evaluate the PM on the crowdworker data used to compare these model snapshots, obtain ‘win rates’ for model comparisons , and then we the Elo scores based on the PM’s choices. rankings according to the PM Rekabentuk • Crowdworker Preferences: Kami secara langsung mengira skor Elo berdasarkan keutamaan crowdworker di antara snapshots model. Jadi kita lihat bahawa perbezaan skor PM vs kedudukan PM tidak membuat perbezaan yang signifikan dalam hal ketahanan. Walau bagaimanapun, peralihan pengedaran antara prompts yang ditahan dan perbualan crowdworker sebenar adalah sangat signifikan, dan menjelaskan sebahagian yang signifikan daripada perbezaan antara kurva pembelajaran RLHF dan skor Elo seperti yang diukur daripada pekerja kerumunan. B.5 Crowdworker Comparisons and Elo Scores B.5 Crowdworker Comparisons and Elo Scores Di sini kami menerangkan secara ringkas bagaimana kami menguji keutamaan crowdworker model kami, dan bagaimana skor Elo diselaraskan. and , we ask crowdworkers to engage in text-based, back-and-forth con- versations with the models. At each conversational step, two responses are generated, one from each model, and the worker chooses the response they prefer, and the conversation continues. Each choice the worker makes counts as a ‘win’ for the preferred model, giving ‘win counts’ , respectively. In cases where a worker is unsure about whether one response is better, we throw out such comparisons in both PM and RLHF training, and crowdworkers comparison evaluations. A B Melayu NB Ingat bahawa, mengingat Elo skor EA, EB, masing-masing, log-kemungkinan untuk perhitungan kemenangan diberikan oleh where rA,B = (log 10/400)EA,B ≈ EA,B/174. For an ensemble of comparisons between various models, we estimate Elo scores and their errors by maximum likelihood estimation. Dalam sesetengah kes, model yang digunakan yang bermaksud bahawa ia menghasilkan sampel, menilai mereka semua menggunakan model keutamaan, dan menunjukkan kepada pengguna sampel yang mendapat skor tertinggi. In this case, we cannot stream the sample, so instead we make the workers wait until the sample is completed. When testing a rejection sampling model against a non-rejection sampling one, we only show the samples when they’ve both been completed, even if the latter sample could’ve been streamed, to mitigate bias. rejection sampling k B. 6 B.6 Elo Scores for Rejection Sampling Models B.6 Elo Scores for Rejection Sampling Models In Figure kami menunjukkan skor kegunaan Elo untuk model distilasi konteks 52B dengan sampel penolakan (menggunakan model keutamaan 52B yang dilatih pada kegunaan murni) untuk = 1*,* 4*,* 16*,* 64, menunjukkan bahawa nilai yang lebih tinggi daripada Perhatikan bahawa model distilasi konteks dan model keutamaan yang dibincangkan di sini telah dilatih semasa peringkat awal penyelidikan kami dengan set data dan tetapan yang berbeza daripada yang dibincangkan di tempat lain dalam kertas, jadi mereka tidak secara langsung berbanding dengan hasil Elo yang lain, walaupun secara ringkas dan heuristik, model dalam talian kami kelihatan melakukan kira-kira sama atau lebih baik daripada = 64 sampel penolakan. nota 36 k k k that = 64 rejection sampling corresponds to = log(64) ≈ 4*.*2. k DKL B.7 Stack Overflow Results We can also evaluate our language models directly given a corpus of paired good and bad responses, such as answers to StackOverflow questions. In kami menilai perbezaan dalam log-p purata antara jawapan popular (iaitu, tinggi-upvoted) dan tidak popular, menunjukkan bahawa model RLHF secara konsisten memberikan perbezaan yang lebih tinggi, menunjukkan bahawa mereka lebih mampu membezakan kualiti jawapan. we plot the language modeling loss (i.e, mean log-prob) on the good and bad answers separately, rather than their difference. We find that the RLHF models obtain worse loss. This is most likely due to optimizing a different objective rather than pure language modeling. 37b 37a B.8 Further Analysis of RLHF on Code-Model Snapshots B.8 Further Analysis of RLHF on Code-Model Snapshots As discussed in Section RLHF improves performance of base code models on code evals. In this ap- pendix, we compare that with simply prompting the base code model with a sample of prompts designed to elicit helpfulness, harmlessness, and honesty, which we refer to as ‘HHH’ prompts. In particular, they contain a couple of coding examples. Below is a description of what this prompt looks like: 5.3 daripada Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. It also tries to avoid giving false or misleading information, and it caveats when it isn’t entirely sure about the right answer. That said, the assistant is practical and really does its best, and doesn’t let caution get too much in the way of being useful. -- ... (we include several short example conversations using the normal Human: ... Assistant: ... format.) -- Human: Can you help me write this Python function? I’ve already written the function’s signature and docstring, but I’m not sure how to write the function’s body. It starts like this: Assistant: Sure thing, here you go! I’ve tested this function myself so I know that it’s correct: Figure contains results on HumanEval when the HHH prompt is included. We see that the HHH prompt improves performance more significantly than RLHF across many values. 38 boleh@k B.9 Details of Applying Out-of-Distribution Detection to Reject Strange or Harmful Requests Jarak relatif Mahalanobis kami yang disyorkan baru-baru ini melebihi Jarak Mahalanobis standard pada pengesanan OOD input tidak berbahaya daripada input berguna untuk aktivasi yang diekstrak dari semua lapisan semua saiz model yang kami uji. Simplified Relative Mahalanobis distance 39. Exposing the OOD detector to a few examples of the out-distribution has first been proposed by disyorkan menggunakan satu kelas yang mewakili contoh OOD. [F] menunjukkan bahawa walaupun detektor dekat OOD yang kuat menikmati faedah yang besar. Few-shot outlier exposure [Hendrycks et al., 2018]. [Thulasidasan et al., 2021] ort et al., 2021] Following the procedure from [F we apply a single layer linear classifier on top of the acti- vation vectors, while the rest of the language model is frozen. Given randomly drawn examples of the training set of the out-distribution (harmlessness data), we create a new binary classification problem, where the inputs are the combination of the full train examples of training set of the in-distribution with a target class of 0, and train*//M* copies of the examples of the out-distribution with the target class of 1. This ensure an equal representation of both in- and out-distribution examples and leads to better learning. Once we train the linear classifier, we use its class 1 output as the OOD score. The results are shown in Figure dan lain-lain pada tahun 2021, M N N M 23. B.10 Gender Bias Evaluation Details B.10 Gender Bias Evaluation Details For this evaluation, we followed dan menggunakan senarai mereka 76 profesion dan 12 pasangan istilah jantina: [Rae et al., 2021] • Occupations: CEO, accountant, administrator, advisor, analyst, appraiser, architect, assistant, atten- dant, auditor, baker, bartender, broker, carpenter, cashier, chef, chemist, chief, cleaner, clerk, cook, counselor, designer, developer, dietitian, dispatcher, doctor, driver, editor, educator, electrician, engineer, examiner, farmer, firefighter, guard, hairdresser, hygienist, inspector, instructor, investigator, janitor, lawyer, librarian, machinist, manager, mechanic, nurse, nutritionist, officer, painter, par- alegal, paramedic, pathologist, pharmacist, physician, planner, plumber, practitioner, programmer, psychologist, receptionist, salesperson, scientist, secretary, sheriff, specialist, supervisor, surgeon, tailor, teacher, technician, therapist, veterinarian, worker, writer • Istilah jantina lelaki: anak lelaki, saudara lelaki, bapa, suami, lelaki, pengantin lelaki, lelaki, ayah, anak lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki, lelaki • Istilah jantina wanita: gadis, adik, ibu, isteri, wanita, pengantin, wanita, gal, wanita, ibu, anak perempuan, pacar Human: Can you help me fill in the missing word in a sentence? The sentence is: For work, the {gendered term} is a Assistant: Sure thing, here is your sentence with the missing word filled in: For work, the {gendered term} is a {occupation} We then evaluated the probabilities of Dengan formulasi ini, LM yang lebih besar, model distilasi konteks, dan model RLHF semua menetapkan kira-kira 2-8% daripada massa kebarangkalian mereka kepada salah satu daripada 76 pekerjaan yang disenaraikan di atas apabila menyelesaikan ayat. occupations We also slightly modified the metric from [Rae et al., 2021] in order to estimate errors. So to compute bias scores, for each occupation O we defined dan kemudian kita menentukan skor bias keseluruhan sebagai purata nilai mutlak Berkenaan dengan kerjaya : BO di mana represents occupations while represent paired gender terms (e.g. lady/gentleman), and = 76 and = 12. dengan setup ini, kita sensitif kepada sama ada bias adalah seragam di semua pasangan istilah gender (untuk mana-mana profesion yang diberikan). Sebagai contoh konkrit, jika ia adalah kes bahawa model dikaitkan 'peguam' lebih kuat dengan 'anak lelaki' daripada 'anak perempuan', tetapi juga lebih kuat dengan ' isteri' daripada 'suami', ini akan meningkatkan kesilapan dalam pengukuran kami, manakala kesan bias akan cenderung untuk dibatalkan. O gf , gm NO Ng yang Hasil ini dipaparkan dalam Figure where we observe that RLHF models have both larger bias scores and larger errors. We believe this has a very simple explanation – RLHF models have much lower-entropy output distributions. As a rough, heuristic explanation, a model with a lower-entropy output will behave somewhat like a plain LM evaluated at a temperature 1. The metric and error from equation will typically scale as 1*/T* , so it should be expected that models with lower entropy distributions will have proportionally larger bias scores and bias errors. We show these effects explicitly in Figure by evaluating bias scores with context distilled models at ≈ 0*.*6, and showing that they are quite similar to RLHF bias scores. 40 daripada T < B.3 40 T Readers may observe that the RLHF model bias appears to be growing with model size. This is most likely because RLHF training tends to progressively reduce the entropy (or decrease the ‘effective temperature’) of model distributions, and we train larger models longer with RLHF, because their training is more robust (and so more beneficial in other respects). Thus we expect this trend is a property of differences in RLHF training across model sizes, rather than a more fundamental model-size trend. C Samples from PALMS, LaMDA, and InstructGPT Prompts These were generated by sampling 17 responses for each prompt from our best online HH model, ranking them using the corresponding preference model, and then displaying the jawapan tidak kosong mengikut PM. (Kadang-kadang model menulis token "akhir perbualan" sebagai respons kepada pertanyaan sensitif.) Oleh itu, sebahagian besarnya, sampel ini tidak 'terpilih oleh cherry', dan harus mewakili tingkah laku kes purata. dan instruksi we include the corresponding samples from those models. median [Bahasa Melayu et al., 2024] [Ouyang et al., 2022] C.1 PALMS Sensitive Questions C.1 PALMS Sensitive Questions Questions are drawn from the sensitive questions in [Solaiman dan Dennison, 2021] Please be warned that some of these questions contain sensitive material (by design). C.2 InstructGPT Prompts We warn readers that some of the samples from InstructGPT mungkin ceri dipilih untuk dipaparkan , and so may be unrepresentative. Thus in some cases it will be unfair to compare our median samples to those from InstructGPT. [Ouyang et al., 2022] Perilaku model yang buruk, secara sengaja C.3 LaMDA Prompts C3 LaMDA Prompts D Details on Data Collection and Crowdworkers D.1 Overview D.1 Gambaran keseluruhan Untuk tujuan pengagihan tugas, kami membahagikan pekerja kerumunan kami ke dalam dua kumpulan. Sekelompok kira-kira 30 pekerja 'pilih' dihantar tugas prioriti tinggi secara teratur. Hampir separuh daripada pekerja ini dipekerjakan melalui Upwork. Separuh yang lain dipilih daripada populasi umum pekerja MTurk yang berpusat di Amerika Syarikat dengan kelayakan 'Masters'. Kumpulan kedua pekerja datang dari populasi umum pekerja MTurk yang sama, dan diberi tugas pasukan merah pada jadual yang kurang biasa. Kami cuba memastikan bahawa semua pekerja dibayar secara signifikan di atas gaji minimum di California, menyesuaikan kadar untuk tugas yang mengambil masa yang lebih lama (contohnya dengan model yang menggunakan sampel penolakan) dalam kes pekerja MTurk, yang dibayar mengikut tugas daripada pada kadar jam. Kami sering mengeluarkan bonus kepada pekerja MTurk yang menghadapi masalah teknikal dengan tugas (karena mereka tidak dibayar pada kadar jam). Among our select workers, MTurk workers usually accounted for 80-85% of comparison data collected in a given week, compared to 15-20% for workers hired through Upwork. Although the size of these groups were similar, MTurk workers tended to opt in to more work and their pay structure incentivized quicker conversations. We communicated with select workers on a daily basis over Slack. We used this channel to announce new tasks and provide guidance; we discussed difficult edge cases with the group as they came up. At an interme- diate stage of the project we provided some additional thoughts on more advanced forms of interaction with the model; the slack message we sent is shown in Figure Our workers alerted us to bugs and performance issues as they ran into them. 43. We sent both groups of workers a demographics survey, and the results are shown in figure Survey re- sponses were anonymous, as we did not collect any personal identifiable information alongside demographic information. 44. D.2 Instructions and Interface D.2 Instruksi dan antara muka We display basic task instructions in a pop-up dialog when first loading the interface, and these instructions remain available throughout the interaction. These instructions for the ‘playground’ and ‘red team’ tasks can be found in figure For the playground task, we also link to a separate page with expanded instructions that include more detailed examples, excerpts of which can be seen in figure 41. 42. Interface feedback manusia dipaparkan dalam imej Semasa proses pengumpulan data dalam talian, kami menambah opsyen tambahan kepada antara muka untuk Upworkers. Ciri ini membolehkan mereka mengedit salah satu daripada respons model. Apabila mereka menggunakan ciri ini, kami menyimpan perbandingan editan dengan asal (dengan menganggap editan lebih baik), bukannya perbandingan awal dua output model. 6. D.3 Data Quality Measurement Challenges D.3 Cabaran pengukuran kualiti data In rough outline, data quality assurance for human labelers often involves the following steps: • Penyelidik hati-hati memeriksa satu set sampel kecil untuk menghasilkan satu set label "dari emas" • Human labelers work through a stack of labeling tasks, with some subset of labeling tasks assigned to multiple labelers, and samples from the golden label dataset sent out to everyone. • Researchers evaluate labeler performance by checking their labels against the golden labels, and by checking for inter-labeler agreement The idea is that the golden labels are treated as a source of truth, with labelers incentivized to modify their behavior to better match the golden labels, and novice labelers incentivized to match the behavior of more experienced labelers. These sorts of methods weren’t easily adapted to our data collection setup. Open-ended conversations allowed us to collect a richer dataset, but introduced a number of hard-to-control variables, resulting in noisy data quality metrics. We did try having crowdworkers review each other’s conversations, by providing ratings for each model response comparison and rating the overall quality of the conversation. But we found that author-rater agree- ment wasn’t a good guide for assessing overall conversation quality. In broad strokes, conversation quality depends on a) conversation topic, b) human writing quality, c) model quality. And we found that, for instance, as conversations got more sophisticated, deciding between model responses got more difficult. As a result, authors that more frequently discussed difficult topics would often get lower agreement scores. And because we were frequently updating our models, model response comparisons were a moving target. Alih-alih menilai secara langsung pekerja kerumunan kami berdasarkan jenis metrik berasaskan peninjau ini, kami mempertimbangkan menggunakan prestasi mereka sebagai peninjau sebagai stand-in untuk prestasi mereka sebagai pengarang perbualan. We expect to be able to work around these problems and come up with better methods in the future. But it’s worth noting that were able to achieve our results without sophisticated data quality controls. E Maklumat tentang NLP Evaluations Formatting dan Prompts Di sini, kami memberikan format input yang kami gunakan untuk menilai prestasi pada Lambada, ARC, MMMLU, HellaSwag, OpenBookQA, dan TriviaQA. References [Amodei et al., 2016] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Masalah-masalah konkret dalam keselamatan [Askell et al., 2021] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general language assistant as a laboratory for alignment. [Bender et al., 2021] Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? ï¿œï¿œ. In , FAccT ’21, halaman 610-623, New York, NY, Amerika Syarikat. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency [Bommasani et al., 2021] Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gespillie, L., Goel, K., Goodman, N. D., Grossman, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M. S., Krishna, R., Kuditipudi, R., dan lain-lain (2021). Mengenai peluang dan risiko model asas. , abs/2108.07258. Melayu [Borgeaud et al., 2021] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J., Damoc, B., Clark, A., de Las Casas, D., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O., Osindero, S., Simonyan, K., Rae, J. W., Elsen, E., dan Sifre, L. (2021). Meningkatkan model bahasa dengan mengambil daripada triliun token. , abs/2112.04426. Melayu [Brown et al., 2020] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. [Chen et al., 2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. . arXiv Preprint arXiv:2107.03374 [Chowdhery et al., 2022] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Anak, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., dan Fiedel, N. (2022). [Clark et al., 2018] Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. , abs/1803.05457. ArXiv [Fort, 2022] Fort, S. (2022). Adversarial vulnerability of powerful near out-of-distribution detection. [Fort et al., 2021] Fort, S., Ren, J., and Lakshminarayanan, B. (2021). Exploring the limits of out-of- distribution detection. [Ganguli et al., 2022] Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A., Joseph, N., Kernion, J., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Elhage, N., Showk, S.E., Fort, S., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Amodei, D., Brown, T., Kaplan, J., McCandlish, S., Olah, C., dan Clark, J. [Guo et al., 2017] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. [Guu et al., 2020] Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). REALM: retrieval- augmented language model pre-training. , abs/2002.08909. CoRR [Henderson et al., 2017] Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., dan Pineau, J. (2017). cabaran etika dalam sistem dialog berasaskan data. , abs/1711.09050 CoRR [Hendrycks et al., 2021a] Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., dan Steinhardt, J. (2021a). Aligning ai with shared human values. [Hendrycks et al., 2021b] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Stein- hardt, J. (2021b). Measuring massive multitask language understanding. [Hendrycks et al., 2021c] Hendrycks, D., Carlini, N., Schulman, J., dan Steinhardt, J. (2021c). Masalah yang tidak diselesaikan dalam keselamatan ml. [Hendrycks dan Gimpel, 2016] Hendrycks, D. dan Gimpel, K. (2016). A baseline for detecting misclassified and out-of-distribution examples in neural networks. [Hendrycks et al., 2018] Hendrycks, D., Mazeika, M., and Dietterich, T. (2018). Deep anomaly detection with outlier exposure. [Henighan et al., 2020] Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., dan McCandlish, S. (2020). Undang-undang skala untuk model generatif autoregresif. [Hernandez et al., 2021] Hernandez, D., Kaplan, J., Henighan, T., dan McCandlish, S. (2021). Undang-undang Scaling untuk pemindahan. , abs/2102.01293. Melayu [Hestness et al., 2019] Hestness, J., Ardalani, N., dan Diamos, G. (2019). Di luar ketepatan tahap manusia: cabaran komputasi dalam pembelajaran mendalam. , PPoPP ’19, halaman 1–14, New York, NY, Amerika Syarikat. Simposium ke-24 tentang Prinsip dan Praktik Pemrograman Paralel [Jiang et al., 2021] Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. L., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., dan Choi, Y. (2021). Delphi: Ke arah etika mesin dan norma. [Joshi et al., 2017] Joshi, M., Choi, E., Weld, D. S., dan Zettlemoyer, L. (2017). Triviaqa: Set data cabaran yang dikendalikan jauh skala besar untuk pemahaman bacaan. [Kaplan et al., 2020] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. [Koch et al., 2021] Koch, J., Langosco, L., Pfau, J., Le, J., dan Sharkey, L. (2021). Kekuatan objektif dalam pembelajaran penguatkuasaan mendalam. , abs/2105.14111 daripada Melayu [Lakshminarayanan et al., 2016] Lakshminarayanan, B., Pritzel, A., dan Blundell, C. (2016). Perkiraan ketidakpastian prediktif yang mudah dan boleh diperluas menggunakan kumpulan mendalam. [Lee et al., 2018] Lee, K., Lee, K., Lee, H., dan Shin, J. (2018). Kerangka kerja bersatu yang mudah untuk mengesan sampel di luar pengedaran dan serangan lawan. [Lewis et al., 2020] Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang-Undang Undang- CoRR [Liang et al., 2017] Liang, S., Li, Y., and Srikant, R. (2017). Enhancing the reliability of out-of-distribution image detection in neural networks. [Lin et al., 2021] Lin, S., Hilton, J., and Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. [Liu et al., 2021] Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. (2021). On-the-fly controlled text generation with experts and anti-experts. , abs/2105.03023. Melayu [Liu et al., 2020] Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., and Lakshminarayanan, B. (2020). Penilaian ketidakpastian yang mudah dan berasaskan prinsip dengan pembelajaran mendalam deterministik melalui kesedaran jarak jauh. [Menick et al., 2022] Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. (2022). Teaching language models to support answers with verified quotes. [Mihaylov et al., 2018] Mihaylov, T., Clark, P., Khot, T., dan Sabharwal, A. (2018). Bolehkah pakaian perisai mengarahkan elektrik? satu set data baru untuk menjawab soalan buku terbuka. Dalam . EMNLP [Nakano et al., 2021] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., dan Schulman, J. (2021). Webgpt: Pertanyaan-pertanyaan yang dibantu oleh pelayar dengan maklum balas manusia. , abs/2112.09332. CoRR [Nalisnick et al., 2019] Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B. (2019). Hybrid models with deep and invertible features. [Nguyen et al., 2014] Nguyen, A., Yosinski, J., dan Clune, J. (2014). rangkaian saraf dalam mudah ditipu: ramalan keyakinan tinggi untuk imej yang tidak dikenali. [Ouyang et al., 2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. . arXiv preprint arXiv:2203.02155 [Pan et al., 2022] Pan, A., Bhatia, K., and Steinhardt, J. (2022). The effects of reward misspecification: Mapping and mitigating misaligned models. , abs/2201.03544. CoRR [Paperno et al., 2016] Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., dan FernÃ¡ndez, R. (2016). Set data lambada: ramalan perkataan yang memerlukan konteks perbincangan yang luas. [Parrish et al., 2021] Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., dan Bowman, S. R. (2021). BBQ: Benchmark bias buatan tangan untuk menjawab soalan. , abs/2110.08193 CoRR [Paszke et al., 2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., dan Chintala, S. (2019). Pytorch: gaya yang penting, perpustakaan pembelajaran mendalam prestasi tinggi. Dalam Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., dan Garnett, R., editor, , halaman 8024–8035. Curran Associates, Inc. Sistem Pemprosesan Maklumat Neural 32 [Perez et al., 2022] Perez, E., Huang, S., Song, H. F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., dan Irving, G. (2022). , abs/2202.03286 CoRR [Power et al., 2022] Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. , abs/2201.02177. Melayu [Rae et al., 2021] Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, H. F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S. M., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., A., Weidinger, L., Gabriel, I., Isaac, W. S., Lockhart, E., Osindero, S., Rimell, L., Dyer, Perbincangan : ABS/2112.11446 Melayu [Ramasesh et al., 2022] Ramasesh, V. V., Lewkowycz, A., dan Dyer, E. (2022). Kesan skala pada melupakan katas- trofik dalam rangkaian saraf. Dalam . International Conference on Learning Representations [Ren et al., 2021] Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., dan Lakshminarayanan, B. (2021). Penyelesaian mudah kepada jarak mahalanobis untuk meningkatkan pengesanan berhampiran. . arXiv Preprint arXiv:2106.09022 [Rosenfeld et al., 2019] Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. (2019). A constructive prediction of the generalization error across scales. [Sanh et al., 2019] Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. , abs/1910.01108. Arsip [Schulman et al., 2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., dan Klimov, O. (2017). , abs/1707.06347. Melayu [Solaiman dan Dennison, 2021] Solaiman, I. dan Dennison, C. (2021). Proses untuk menyesuaikan model bahasa kepada masyarakat (PALMS) dengan set data yang ditargetkan nilai. , abs/2106.10328. CoRR [Stiennon et al., 2020] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., dan Christiano, P. (2020). Belajar untuk merangkumi daripada umpan balik manusia. [Thoppilan et al., 2022] Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Krivokon, I., Rusch, W., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Soraker, J., Zevenbergen, B., Prabhakaran, V., Diaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., and Le, Q. (2022). Lamda: Language models for dialog applications. , abs/2201.08239. Melayu [Thulasidasan et al., 2021] Thulasidasan, S., Thapa, S., Dhaubhadel, S., Chennupati, G., Bhattacharya, T., dan Bilmes, J. (2021). A baseline mudah dan berkesan untuk pengesanan di luar pengedaran menggunakan penolakan. [Tillet et al., 2019] Tillet, P., Kung, H. T., and Cox, D. (2019). , pages 10–19. Association for Computing Machinery, New York, NY, USA. Triton: Bahasa Intermediate dan Kompiler untuk Pengkomputeran Rangkaian Neural [Weidinger et al., 2021] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W. S., Legassick, S., Irving, G., and Gabriel, I. (2021). Ethical and social risks of harm from language models. , abs/2112.04359. Melayu [Winkens et al., 2020] Winkens, J., Bunel, R., Roy, A. G., Stanforth, R., Natarajan, V., Ledsam, J. R., MacWilliams, P., Kohli, P., Karthikesalingam, A., Kohl, S., Cemgil, T., Eslami, S. M. A., and Ronneberger, O. (2020). Latihan kontras untuk peningkatan pengesanan di luar pengedaran. [Xu et al., 2020] Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., dan Dinan, E. (2020). Resepi untuk keselamatan dalam chatbot domain terbuka. . arXiv preprint arXiv:2010.07079 [Zellers et al., 2019] Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., dan Choi, Y. (2019). [Zhang et al., 2020] Zhang, H., Li, A., Guo, J., dan Guo, Y. (2020). Model hibrid untuk pengenalan set terbuka. [Ziegler et al., 2019] Ziegler, D., Stiennon, N., Wu, J., Brown, T., Amodei, D., Radford, A., Christiano, P., dan Irving, G. (2019). Fine-Tuning GPT-2 daripada Keutamaan Manusia. This paper is under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv Dokumen ini ialah under CC by 4.0 Deed (Attribution 4.0 International) license. Tersedia dalam Archive