Caspar.AI

casparai

GPT-4 vs. Humans: Validating AI Judgment in Language Model Training

Authors:
(1) TIMNIT GEBRU, Black in AI;
(2) JAMIE MORGENSTERN, University of Washington;
(3) BRIANA VECCHIONE, Cornell University;
(4) JENNIFER WORTMAN VAUGHAN, Microsoft Research;
(5) HANNA WALLACH, Microsoft Research;
(6) HAL DAUMÉ III, Microsoft Research; University of Maryland;
(7) KATE CRAWFORD, Microsoft Research. Table of Links 1 Introduction 1.1 Objectives 2 Development Process 3 Questions and Workflow 3.1 Motivation 3.2 Composition 3.3 Collection Process 3.4 Preprocessing/cleaning/labeling 3.5 Uses 3.6 Distribution 3.7 Maintenance 4 Impact and Challenges Acknowledgments and References Appendix Acknowledgments We thank Peter Bailey, Emily Bender, Yoshua Bengio, Sarah Bird, Sarah Brown, Steven Bowles, Joy Buolamwini, Amanda Casari, Eric Charran, Alain Couillault, Lukas Dauterman, Leigh Dodds, Miroslav Dudík, Michael Ekstrand, Noémie Elhadad, Michael Golebiewski, Nick Gonsalves, Martin Hansen, Andy Hickl, Michael Hoffman, Scott Hoogerwerf, Eric Horvitz, Mingjing Huang, Surya Kallumadi, Ece Kamar, Krishnaram Kenthapadi, Emre Kiciman, Jacquelyn Krones, Erik Learned-Miller, Lillian Lee, Jochen Leidner, Rob Mauceri, Brian Mcfee, Emily McReynolds, Bogdan Micu, Margaret Mitchell, Sangeeta Mudnal, Brendan O’Connor, Thomas Padilla, Bo Pang, Anjali Parikh, Lisa Peets, Alessandro Perina, Michael Philips, Barton Place, Sudha Rao, Jen Ren, David Van Riper, Anna Roth, Cynthia Rudin, Ben Shneiderman, Biplav Srivastava, Ankur Teredesai, Rachel Thomas, Martin Tomko, Panagiotis Tziachris, Meredith Whittaker, Hans Wolters, Ashly Yeo, Lu Zhang, and the attendees of the Partnership on AI’s April 2019 ABOUT ML workshop for valuable feedback. References [1] Don A Andrews, James Bonta, and J Stephen Wormith. 2006. The recent past and near future of risk and/or need assessment. Crime & Delinquency 52, 1 (2006), 7–27. [2] Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6 (2018), 587–604. [3] Anant P. Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J. Elmore, Samuel Madden, and Aditya G. Parameswaran. 2014. DataHub: Collaborative Data Science & Dataset Version Management at Scale. CoRR abs/1409.0798 (2014). [4] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Advances in Neural Information Processing Systems (NeurIPS). [5] Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*). 77–91. [6] Yang Trista Cao and Hal Daumé. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). abs/1910.13913. [7] Yang Trista Cao and Hal Daumé, III. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). [8] James Cheney, Laura Chiticariu, and Wang-Chiew Tan. 2009. Provenance in databases: Why, how, and where. Foundations and Trends in Databases 1, 4 (2009), 379–474. [9] Kasia Chmielinski, Sarah Newman, Matt Taylor, Josh Joseph, Kemi Thomas, Jessica Yurkofsky, and Yue Chelsea Qiu. 2020. The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence. In NeurIPS Workshop on Dataset Curation and Security. [10] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC : Question Answering in Context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [11] Glennda Chui. 2017. Project will use AI to prevent or minimize electric grid failures. [Online; accessed 14-March-2018]. [12] Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazonscraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G [13] Clare Garvie, Alvaro Bedoya, and Jonathan Frankle. 2016. The Perpetual Line-Up: Unregulated Police Face Recognition in America. Georgetown Law, Center on Privacy & Technology, New Jersey Ave NW, Washington, DC. [14] Michael Hind, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, Karthikeyan Natesan Ramamurthy, Alexandra Olteanu, and Kush R. Varshney. 2018. Increasing Trust in AI Services through Supplier’s Declarations of Conformity. CoRR abs/1808.07261 (2018). [15] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miroslav Dudík, and Hanna M. Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In 2019 ACM CHI Conference on Human Factors in Computing Systems. [16] Gary B Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49. University of Massachusetts Amherst. [17] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. 2017. OpenImages: A public dataset for large-scale multi-label and multi-class image classification. [18] Tom CW Lin. 2012. The new investor. UCLA Law Review 60 (2012), 678. [19] G Mann and C O’Neil. 2016. Hiring Algorithms Are Not Neutral. https://hbr.org/2016/12/hiring-algorithms-are-not-neutral. [20] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*). 220–229. [21] Mary Catherine O’Connor. 2017. How AI Could Smarten Up Our Water System. [Online; accessed 14-March-2018]. [22] Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. 271. [23] Ismaïla Seck, Khouloud Dahmane, Pierre Duthon, and Gaëlle Loosli. 2018. Baselines and a datasheet for the Cerema AWP dataset. CoRR abs/1806.04016 (2018). http://arxiv.org/abs/1806.04016 [24] Doha Suppy Systems. 2017. Facial Recognition. [Online; accessed 14-March-2018]. [25] World Economic Forum Global Future Council on Human Rights 2016–2018. 2018. How to Prevent Discriminatory Outcomes in Machine Learning. https://www.weforum.org/whitepapers/how-to-prevent-discriminatory-outcomes-inmachine-learning. [26] Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing This paper is available on arxiv under CC 4.0 license. Authors: (1) TIMNIT GEBRU, Black in AI; (2) JAMIE MORGENSTERN, University of Washington; (3) BRIANA VECCHIONE, Cornell University; (4) JENNIFER WORTMAN VAUGHAN, Microsoft Research; (5) HANNA WALLACH, Microsoft Research; (6) HAL DAUMÉ III, Microsoft Research; University of Maryland; (7) KATE CRAWFORD, Microsoft Research. Authors: Authors: (1) TIMNIT GEBRU, Black in AI; (2) JAMIE MORGENSTERN, University of Washington; (3) BRIANA VECCHIONE, Cornell University; (4) JENNIFER WORTMAN VAUGHAN, Microsoft Research; (5) HANNA WALLACH, Microsoft Research; (6) HAL DAUMÉ III, Microsoft Research; University of Maryland; (7) KATE CRAWFORD, Microsoft Research. Table of Links 1 Introduction 1 Introduction 1.1 Objectives 1.1 Objectives 2 Development Process 2 Development Process 3 Questions and Workflow 3 Questions and Workflow 3.1 Motivation 3.1 Motivation 3.2 Composition 3.2 Composition 3.3 Collection Process 3.3 Collection Process 3.4 Preprocessing/cleaning/labeling 3.4 Preprocessing/cleaning/labeling 3.5 Uses 3.5 Uses 3.6 Distribution 3.6 Distribution 3.7 Maintenance 3.7 Maintenance 4 Impact and Challenges 4 Impact and Challenges Acknowledgments and References Acknowledgments and References Appendix Appendix Acknowledgments We thank Peter Bailey, Emily Bender, Yoshua Bengio, Sarah Bird, Sarah Brown, Steven Bowles, Joy Buolamwini, Amanda Casari, Eric Charran, Alain Couillault, Lukas Dauterman, Leigh Dodds, Miroslav Dudík, Michael Ekstrand, Noémie Elhadad, Michael Golebiewski, Nick Gonsalves, Martin Hansen, Andy Hickl, Michael Hoffman, Scott Hoogerwerf, Eric Horvitz, Mingjing Huang, Surya Kallumadi, Ece Kamar, Krishnaram Kenthapadi, Emre Kiciman, Jacquelyn Krones, Erik Learned-Miller, Lillian Lee, Jochen Leidner, Rob Mauceri, Brian Mcfee, Emily McReynolds, Bogdan Micu, Margaret Mitchell, Sangeeta Mudnal, Brendan O’Connor, Thomas Padilla, Bo Pang, Anjali Parikh, Lisa Peets, Alessandro Perina, Michael Philips, Barton Place, Sudha Rao, Jen Ren, David Van Riper, Anna Roth, Cynthia Rudin, Ben Shneiderman, Biplav Srivastava, Ankur Teredesai, Rachel Thomas, Martin Tomko, Panagiotis Tziachris, Meredith Whittaker, Hans Wolters, Ashly Yeo, Lu Zhang, and the attendees of the Partnership on AI’s April 2019 ABOUT ML workshop for valuable feedback. References [1] Don A Andrews, James Bonta, and J Stephen Wormith. 2006. The recent past and near future of risk and/or need assessment. Crime & Delinquency 52, 1 (2006), 7–27. [2] Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6 (2018), 587–604. [3] Anant P. Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J. Elmore, Samuel Madden, and Aditya G. Parameswaran. 2014. DataHub: Collaborative Data Science & Dataset Version Management at Scale. CoRR abs/1409.0798 (2014). [4] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Advances in Neural Information Processing Systems (NeurIPS). [5] Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*). 77–91. [6] Yang Trista Cao and Hal Daumé. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). abs/1910.13913. [7] Yang Trista Cao and Hal Daumé, III. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of the Conference of the Association for Computational Linguistics (ACL). [8] James Cheney, Laura Chiticariu, and Wang-Chiew Tan. 2009. Provenance in databases: Why, how, and where. Foundations and Trends in Databases 1, 4 (2009), 379–474. [9] Kasia Chmielinski, Sarah Newman, Matt Taylor, Josh Joseph, Kemi Thomas, Jessica Yurkofsky, and Yue Chelsea Qiu. 2020. The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence. In NeurIPS Workshop on Dataset Curation and Security. [10] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC : Question Answering in Context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [11] Glennda Chui. 2017. Project will use AI to prevent or minimize electric grid failures. [Online; accessed 14-March-2018]. [12] Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazonscraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G [13] Clare Garvie, Alvaro Bedoya, and Jonathan Frankle. 2016. The Perpetual Line-Up: Unregulated Police Face Recognition in America. Georgetown Law, Center on Privacy & Technology, New Jersey Ave NW, Washington, DC. [14] Michael Hind, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, Karthikeyan Natesan Ramamurthy, Alexandra Olteanu, and Kush R. Varshney. 2018. Increasing Trust in AI Services through Supplier’s Declarations of Conformity. CoRR abs/1808.07261 (2018). [15] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miroslav Dudík, and Hanna M. Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In 2019 ACM CHI Conference on Human Factors in Computing Systems. [16] Gary B Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49. University of Massachusetts Amherst. [17] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. 2017. OpenImages: A public dataset for large-scale multi-label and multi-class image classification. [18] Tom CW Lin. 2012. The new investor. UCLA Law Review 60 (2012), 678. [19] G Mann and C O’Neil. 2016. Hiring Algorithms Are Not Neutral. https://hbr.org/2016/12/hiring-algorithms-are-not-neutral. [20] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*). 220–229. [21] Mary Catherine O’Connor. 2017. How AI Could Smarten Up Our Water System. [Online; accessed 14-March-2018]. [22] Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. 271. [23] Ismaïla Seck, Khouloud Dahmane, Pierre Duthon, and Gaëlle Loosli. 2018. Baselines and a datasheet for the Cerema AWP dataset. CoRR abs/1806.04016 (2018). http://arxiv.org/abs/1806.04016 [24] Doha Suppy Systems. 2017. Facial Recognition. [Online; accessed 14-March-2018]. [25] World Economic Forum Global Future Council on Human Rights 2016–2018. 2018. How to Prevent Discriminatory Outcomes in Machine Learning. https://www.weforum.org/whitepapers/how-to-prevent-discriminatory-outcomes-inmachine-learning. [26] Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Using Language Models to Simulate Human Samples: Acknowledgments and References

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 5 CONCLUSION Previous work has shown that adapting LLMs with RLHF helps them generate outputs that are preferred by humans over the supervised fine-tuned counterpart. However, recent work has also pointed out that simply training LLMs to abide by the preference of the general may result in ignoring individual preferences and values. In this work, we provide the first steps to tackle this issue by proposing Reinforcement Learning from Personalized Human Feedback as a multi-objective problem so that LLMs can be aligned to follow conflicting preferences. We propose a promising method called P-SOUPS that is able to composite models trained on single objectives on the fly during inference. We also highlight the scalability of P-SOUPS by showing that it scales linearly, instead of exponentially like the MORL baseline, with regards to the number of new preferences, which is required to provide true personalization to individual users. ACKNOWLEDGMENTS Thanks to Minyoung Hwang, Sungdong Kim, Tim Dettmers, Yoonjoo Lee, and Margaret Li for helpful feedback and discussion. REFERENCES Xiang Ao, Xiting Wang, Ling Luo, Ying Qiao, Qing He, and Xing Xie. PENS: A dataset and generic framework for personalized news headline generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 82–92, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.7. URL https: //aclanthology.org/2021.acl-long.7. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.05862, 2022a. URL https://api.semanticscholar.org/ CorpusID:248118878. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jer´ emy Scheurer, Javier ´ Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias, Idan Szpektor, Craig Boutilier, et al. Dynamic planning in openended dialogue using reinforcement learning. arXiv preprint arXiv:2208.02294, 2022. Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378, 2022. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. arXiv preprint arXiv:2305.08283, 2023. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1, 2023. Conor F Hayes, Roxana Radulescu, Eugenio Bargiacchi, Johan K ˘ allstr ¨ om, Matthew Macfarlane, ¨ Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=nZeVKeeFYf9. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. ArXiv, abs/2307.13269, 2023. URL https://api.semanticscholar.org/CorpusID:260155012. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ICLR, 2022. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 14702–14729. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/jang23a. html. Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. arXiv preprint arXiv:2305.13735, 2023. Hannah Rose Kirk, Bertie Vidgen, Paul Rottger, and Scott A Hale. Personalisation within bounds: A ¨ risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint arXiv:2303.05453, 2023. Noriyuki Kojima, Alane Suhr, and Yoav Artzi. Continual learning for grounded instruction generation by observing human following behavior. Transactions of the Association for Computational Linguistics, 9:1303–1319, 2021. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306, 2022. Pan Li and Alexander Tuzhilin. Towards controllable and personalized review generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3237–3245, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1319. URL https://aclanthology.org/D19-1319. Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. Generating personalized recipes from historical user preferences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5976–5982, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1613. URL https://aclanthology.org/D19-1613. Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. Training mil- ´ lions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2775–2779, Brussels, Belgium, OctoberNovember 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1298. URL https://aclanthology.org/D18-1298. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332, 2021a. URL https://api.semanticscholar.org/CorpusID:245329531. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021b. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022a. URL https://api.semanticscholar.org/ CorpusID:246426909. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022b. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. arXiv preprint arXiv:2205.06266, 2022. Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv preprint arXiv:2306.04488, 2023. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406, 2023. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548, 2023. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In International Conference on Machine Learning, pp. 8905–8915. PMLR, 2020. David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306.17492, 2023. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. Alane Suhr and Yoav Artzi. Continual learning for instruction following from realtime feedback. arXiv preprint arXiv:2212.09710, 2022. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee´ Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and ` efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. Humanaligned artificial intelligence is a multiobjective problem. Ethics and Information Technology, 20:27–40, 2018. Kristof Van Moffaert and Ann Nowe. Multi-objective reinforcement learning using sets of pareto ´ dominating policies. The Journal of Machine Learning Research, 15(1):3483–3512, 2014. Kristof Van Moffaert, Madalina M Drugan, and Ann Nowe. Scalarized multi-objective reinforce- ´ ment learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 191–199. IEEE, 2013. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023. Mitchell Wortsman, Suchin Gururangan, Shen Li, Ali Farhadi, Ludwig Schmidt, Michael Rabbat, and Ari S Morcos. lo-fi: distributed fine-tuning without communication. arXiv preprint arXiv:2210.11948, 2022a. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965–23998. PMLR, 2022b. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021a. Yuwei Wu, Xuezhe Ma, and Diyi Yang. Personalized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1956–1970, 2021b. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693, 2023. Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, pp. 10607–10616. PMLR, 2020. Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5180–5197, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.356. URL https: //aclanthology.org/2022.acl-long.356. Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243, 2018. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. Personalized dialogue generation with diversified traits. arXiv preprint arXiv:1901.09672, 2019. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. A DETAILS OF EVALUATION SETUP We use a modified version of the GPT4 annotation prompt used by Dubois et al. (2023). We modify the criteria to perform the pairwise evaluation from general to a single preference dimension. We also provide 3 demonstrations: one scenario where there is a tie because both responses do not contain any notion of the preference (e.g. both responses do not show any signs of friendliness), one scenario where there is a clear winner, and one scenario where they are both good, but one is better than the other. Simulated pairwise evaluation We recruited 24 crowd workers for our human evaluation. Figure 5 shows the interface used for human evaluation. We consider both the ‘Tie’ and ‘Both are bad’ options to be Ties. B CRITERIA-WISE EVALUATION The criteria-wise win rate (%) across all the methods are shown in Figure 6. The criteria-wise win rate was calculated by getting the average win-rate of the preference combinations that contained the specific preference dimension. For example, when calculating the criteria-wise win rate of ‘Elementary’, we got the average win-rate of the preference combinations that contained the ‘Elementary’ preference, which includes AAA, AAB, ABA, and ABB. C THE FULL LIST OF EVALUATION PROMPTS The full list of evaluation prompts used in our experiments are provided in Table 5 D DETAILED RESULTS FOR HUMAN EVALUATION AND GPT-4 EVALUATION We provide detailed results (win / loss / tie for each of the preference combinations for our main experimental results. Table 6 shows the GPT4 evaluation and Table 7 shows the human evaluation results. E EXAMPLES OF P-SOUPS TEXT GENERATIONS Table 8 shows empirical examples of the text generated from each preference combination of the 16 preference combination experiments for the same prompt.

Personalized Soups: LLM Alignment Via Parameter Merging - Conclusion & References

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 4 EXPERIMENTS 4.1 BASELINE METHODS In this subsection, we provide details of the single-objective baseline methods we implement. The summary of the key component differences in comparison with our proposed methods is provided in Table 2. As the most simple baseline, we simply utilize the base Tulu-7B model to generate responses without providing it any notion of preferences. During the evaluation, we use the same response to evaluate on the 8 different preference combinations. Vanilla Baseline (VB) We perform RLHF in the traditional manner where GPT-4 labels which response is generally better, train a reward model using the pairwise feedback data, and use the reward model to adapt the policy model with PPO training. The same 10k instances from Dtrain are used for RLHF. Reinforcement Learning from Human Feedback (RLHF) Next, we observe how far the instruction-tuned base LM can integrate multiple preference combinations by simply prompting for the preferences without any additional training Preference Prompting (PP) For a competitive baseline, we utilize the positive candidate selected by GPT-4 as the output for imitation learning, which is essentially performing rejection sampling (Nakano et al., 2021a) that uses GPT-4 as the reward model for selecting golden responses from the distribution of responses. We append the individual preference prompt with instances from Dtrain and multitask train the Tulu-7B model across all six individual preferences. This method also allows distilling the outputs of Tulu-30B for training Tulu-7B. Multi-task Training (MT) 4.2 EXPERIMENTAL DETAILS For both the reward model and policy training, we limit ourselves to going through Dtrain only once (1 epoch). In the initial exploration stage, the end performance for the policy model did not improve even if we trained the reward model for longer epochs. For policy model training, we utilize our evaluation dataset (50 prompts) to get the average reward and chose the policy model checkpoint that showed the highest average reward on the evaluation set for our final evaluation. We utilize LoRA (Hu et al., 2022) for both the reward model and policy model training. The detailed hyperparameters for the reward model and policy model training are provided in our github repository [5] . 4.3 MAIN RESULTS Table 3 and 4 show the results of doing all possible pairwise comparisons across the methods using GPT-4 and humans as judges, respectively. Note that the win rate of each battle is calculated using the aggregated win rate explained in Section 3.3. Each individual preference combination results are shown in Appendix D. We also show the average criteria-wise win rate instead of the aggregated win rate across all of the methods in Appendix B. The first thing to note is that there is a limitation to the extent prompting (PP) can integrate multiple preferences. This means that specific training for integrating the multiple preferences is necessary to composite them together. Next, we can see that supervised fine-tuning (RS) underperforms MORLbased methods, which is consistent with prior work that also showed the advantage of RL-based approaches when aligning LLMs with human feedback compared to its supervised-finetuning counterpart. Finally, while P-MORL and P-SOUPS both outperform other methods on average, there exists a discrepancy between the simulated and human evaluation; P-SOUPS has the highest average win rate in GPT-4 evaluation while P-MORL has the highest in human evaluation. Nonetheless,P-SOUPS is able to show superior performance in comparison to baseline methods and competitive performance to P-MORL. In previous parameter merging literature, multi-task fine-tuning (RS) used to be considered the upper bound for compositional abilities through parameter merging (Don-Yehiya et al., 2022). However, in our scenario, we can see that parameter merging (P-SOUPS) is able to outperform multitask fine-tuning, showing promising results for parameter merging not only as a distributed multitask finetuning method but a method that can result in superior performance than multitask training. One might still wonder about the general ‘helpfulness’ capabilities of models that are trained to be tailored to multiple preferences. In Figure 3, we first show the average pairwise win rate from Table 3 in green. Next, we instead ONLY perform pairwise comparisons with the unseen objective ‘helpfulness’ (ask GPT-4 which model response they generally prefer better) and report the average pairwise win rate in red. Trade-off Between General Helpfulness RLHF performs the best in this scenario, which shows that there is no free lunch; while the objective of RLHF was to provide model responses that are generally preferred (highly correlated with ‘helpfulness’), the other methods were prompted/trained to be optimized towards the personalized preference aspects, possibly deviating away from general helpfulness. While RS, P-MORL, and P-SOUPS are able to retain similar performance in terms of helpfulness compared to the initial instructiontuned model (VB), we observe that prompting (PP) significantly underperforms compared to other methods which also highlights the limitation of simply prompting base/instruction-tuned models for personalized preferences and shows the need for specialized training methods for personalization. 4.4 SCALING TO NEW PREFERENCES While we explore 6 distinct preferences in this work, we are still limited in doing ‘declarative’ personalization; that is, the individual preferences have been pre-defined to measure the performance of different methodologies. However, in the real world, individuals may not be bound by predefined preferences. Furthermore, people’s preferences might change over time, which requires continual learning of new preferences. This means that we may be required to train infinite numbers of preferences to be truly personalized to individuals’ preferences. Considering this aspect, the scalability of methods becomes a critical factor in implementing RLPHF in real-world scenarios. In order to compare the scalability of P-MORL and P-SOUPS, we add two new preferences (in addition to the ones in Table 1 to the STYLE dimensions: (P3C) “Generate/Choose a response (that answers) in a sassy manner.” and (P3D) “Generate/Choose a response (that answers) in a sarcastic manner.”, which results in a total of 16 (2 × 2 × 4) unique preference combinations. We re-train P-MORL on the 16 new preference combinations and only train two new policy models for integrating P-SOUPS. The simulated win-rate between P-MORL and P-SOUPS on each of the original preference combinations (53.04% win rate of P-SOUP over P-MORL in Table 3 decomposed into each preference combinations) and the 16 new preference combinations are shown in Figure 4. As shown in the figure, P-SOUPS shows competitive performance compared to P-MORL while being much more efficient considering that it (1) did not have to observe all 16 possible preference combinations and (2) did not have to re-train on the previous preferences, but just train two new policies each for the new preference in a modular manner and merge their parameters on-the-fly during inference. Considering that P-MORL is bounded by O(2n) while P-SOUPS is bounded by O(n) where n is the total number of preferences (assuming there are two unique preferences for each dimension), we assert that P-SOUPS allows tackling RLPHF to be feasible. [5] https://github.com/joeljang/RLPHF

Personalized Soups: LLM Alignment Via Parameter Merging - Experiments

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 3 REINFORCEMENT LEARNING FROM Personalized HUMAN FEEDBACK Collecting Conflicting Pairwise Feedback We utilize Tulu-7B LM (Wang et al., 2023), a model that uses LLaMA-7B (Touvron et al., 2023) as a base model and is instruction tuned on a mixture of open-source instruction-tuning datasets, as the base model for our experiments. We utilize 10k prompt instances from GPT4-Alpaca (Peng et al., 2023), one of the datasets used to train Tulu-7B, as our instruction dataset Dtrain to generate rollouts and collect pairwise feedback data. We also use the same Dtrain during Proximal Policy Optimization (PPO) training (Schulman et al., 2017) of Tulu-7B. Following previous work, we simulate human annotators with GPT-4 for collecting large-scale pairwise feedback data (Bai et al., 2022b; Dubois et al., 2023)—but note that our evaluations are validated with (smaller-scale) human preference data collected from crowdworkers. While Dubois et al. (2023) mostly simulates GPT-4 and other LLMs to choose which is generally a better response between two candidate responses, we provide GPT-4 with a single preference (full list shown in Table 1) to decide which is a better response. We also provide the same preference criteria via additional prompts during the rollout generation of the two candidate responses; we use Tulu-30B for the rollout generation while the actual policy model we train is Tulu-7B for our main experimental setup, making our experimental setting an off-policy training set-up. While we have feedback on which of the two model responses is more aligned with a single preference via GPT-4 annotation, utilizing only two positive pairs during reward model training was empirically shown to be less robust during the PPO training. Instead, we train our reward model on multiple comparisons (Song et al., 2023; Kim et al., 2023) by including a neutral response and a negative response as shown in Figure 2. Specifically, the reward model is provided with four different comparisons for a single prompt during training: positive 1 > positive 2 (decided by GPT-4), positive > neutral, positive > negative, and neutral > negative. The positive response when compared with the neutral and the negative response is chosen randomly. This allows the reward model to be exposed to different granularity of the specific preference and give scores accordingly during PPO training. We explore (1) training a single reward model in a multitask fashion that leverages the preference prompts during inference to give distinct rewards according to each preference and (2) training multiple reward models, each tailored to the distinct preference. Reward Model Training 3.2 MULTI-OBJECTIVE REINFORCEMENT LEARNING (MORL) The MORL problem can be denoted as: 3.3 MULTIFACETED EVALUATION For evaluation, we manually filter out 50 instances from the Koala evaluation (Geng et al., 2023) that require open-ended generations. We also modified some of the prompts so that the evaluation prompts do not contain any elements requiring individual preferences (e.g., removing the phrase asking for a elementary-level response from the original prompt since we want to test the LLM to generate a expert-level response). The full list of evaluation prompts used for our experiments is shown in Appendix C. In our evaluation setup, we simulate users to have a unique combination of preferences, each from the three preference dimensions (Expertise, Informativeness, Style) in Table 1, which equates to 8 unique preference combinations (examples shown in Figure 1). We get the average win rate across the simulated 8 preference combinations for our final evaluation. We use a variant of the AlpacaFarm evaluation framework for simulated (GPT4) evaluation and hire 24 crowdworkers for human evaluation. Details of human evaluation are provided in Appendix A. Evaluation [3] P1, P2, P3 each represents preference prompts from each preference dimension in Table 1. For one example, one unique combination might be P1A + P2B + P3A (ABA) where the combined objective for the response needs to be elementary level, informative, and friendly. [4] Empirically, utilizing a single reward model instead of multiple reward models led to better performance. We hypothesize this is due to the problem of normalizing signals from different reward models (Hayes et al., 2022), which is known to be a nontrivial problem in MORL.

Personalized Soups: LLM Alignment Via Parameter Merging - Personalized Human Feedback

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 2 RELATED WORK Incorporating human preference feedback into a reward model, and subsequently optimizing a language model to output text that reward model scores highly with an RL algorithm, has been shown to result in language models that generate outputs humans generally prefer (Ouyang et al., 2022b). This process has been applied to summarization (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021a), answering questions with long-form answers using text retrieved from the web (Nakano et al., 2021b; Menick et al., 2022), generating engaging responses in a dialogue settings (Thoppilan et al., 2022; Cohen et al., 2022) and following human instructions (Kojima et al., 2021; Suhr & Artzi, 2022; Kim et al., 2023). Aligning Language Models To Human Preferences However, the standard RLHF setup commonly addressed in prior work assumes a reward model that accounts only for average annotator preference, i.e., the fact that different users may desire different outputs, even for the same prompt, is ignored Casper et al. (2023). Individual preferences can vary not only on aesthetic axes, but also on semantics. For example, Santurkar et al. (2023) use public opinion polling to show that “default” LLM preferences vary in their degree of expressed-opinion alignment with different average opinions among demographic groups.[2] Kirk et al. (2023) defines a taxonomy and policy framework for the alignment of LLMs with personalized feedback. While Wu et al. (2023) performs fine-grained RLHF which is very similar in spirit and allows personalization, our work develops MORL algorithms for scenarios where there are conflicting preferences, not only orthogonal objectives. In this work, we propose formulating LLM personalization as a MORL problem, which was typically studied in decision-making tasks (Hayes et al., 2022) that aims to tackle the problem of simply optimizing by a single, scalar, additive reward function (Sutton & Barto, 2018), which possesses many limitations such as (1) suboptimal solutions due to lack of representation (Hayes et al., 2022), (2) lack of explainability of distinct objectives, and (3) ensuring fair outcomes for multiple participants (Vamplew et al., 2018; Siddique et al., 2020). Multi-objective Reinforcement Learning (MORL) Previous work has aimed to alleviate these problems through novel MORL methods (Van Moffaert et al., 2013; Van Moffaert & Nowe´, 2014; Yang et al., 2019; Xu et al., 2020). Other work aims to solve complex problems such as water management, military purchasing, wind farm control, etc. (Hayes et al., 2022) by converting the single-objective RL problem into a MORL problem. In this work, we convert the problem of aligning LLMs to human preferences into a MORL problem to (1) provide a more optimal solution for each individual, (2) allow users to dynamically choose the distinct objectives they want to optimize, and (3) ensure fairness by allowing preferences that may be in the long-tail to be integrated. Personalization in Natural Language Processing (NLP) has mainly been focused on creating personalized dialogue agents (Zhang et al., 2018; Mazare et al. ´ , 2018; Zheng et al., 2019; Wu et al., 2021b; Xu et al., 2022), where the task is to create chitchat agents that are engaging with distinct personas based on user profile (e.g. gender, age, residence, etc.) or past user history data (e.g. Reddit posts, etc.). Another line of work (Salemi et al., 2023) leverages personalized information to boost performance on specific tasks such as review generation (Li & Tuzhilin, 2019), recipe generation (Majumder et al., 2019), and headline generation (Ao et al., 2021). This line of work requires model providers to make better models utilizing the personal information of the user. In our work, we propose a framework that allows users to choose which preference the language model should prefer, essentially giving control to the user. Personalization in Natural Language Processing Recent work has shown that performing weighted linear interpolation of model parameters leads to the composition of each model ability (Li et al., 2022; Wortsman et al., 2022b;a; Don-Yehiya et al., 2022; Huang et al., 2023). This line of work has led to many interesting applications of model merging such as composing the abilities of expert models that perform different tasks (Ilharco et al., 2022; Jang et al., 2023) and introducing language-specific modules for growing the total capacity of multilingual LMs (Pfeiffer et al., 2022). Parameter Merging Most recently, Rame et al. (2023) proposed to merge policy models that were trained to perform specific tasks such as question answering and summarization using proxy reward models. While they mostly deal with reward models trained on the same data, our proposed MORL methods are an extension of this work that actually deals with diverse reward models trained on multifaceted human feedback to show compositional abilities through parameter merging rather than just ensembling. [2] Feng et al. (2023) suggests that “default” LLM expressed opinions stem directly from the pretraining data.

Personalized Soups: LLM Alignment Via Parameter Merging - Related Work

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. ABSTRACT While Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with general, aggregate human preferences, it is suboptimal for learning diverse, individual perspectives. In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem. Compared to strong single-objective baselines, we show that we can achieve personalized alignment by decomposing preferences into multiple dimensions. These dimensions are defined based on personalizations that are declared as desirable by the user. In this work, we show that they can be efficiently trained independently in a distributed manner and combined effectively post-hoc through parameter merging. [1] 1 INTRODUCTION Reinforcement Learning from Human Feedback (RLHF) (Nakano et al., 2021a; Ouyang et al., 2022a; Bai et al., 2022a; Dubois et al., 2023; Bai et al., 2022b) typically optimizes a policy model that receives training signals from a single reward model that aims to capture the general preferences of a population. In this work, we instead propose Reinforcement Learning from Personalized Human Feedback (RLPHF), a new, multi-objective formulation of the human preference alignment problem, where Large Language Models (LLMs) are trained to be efficiently aligned with a range of different, potentially personalized combinations of human preferences. We model RLPHF as a Multi-Objective Reinforcement Learning (MORL) problem, which allows training the policy model with multiple, conflicting objectives since it aims to vary the importance of each objective during inference. In existing RLHF formulations, pairwise human feedback is collected by asking human annotators to choose which model response is generally better and is used to train a general reward model. This makes implicit assumptions that may not hold for everyone. For example, recent work has shown that LLMs aligned with RLHF prefer verbose output generations (Zheng et al., 2023; Dubois et al., 2023; Wang et al., 2023; Singhal et al., 2023). We aim to support a wider range of multifaceted preferences that are explicitly declared as desirable by the user—giving the user control over the facets of output text they want to see as well as the personal data they wish to reveal to the model. We collect personalized human feedback corresponding to multiple such dimensions, noting that they may also be conflicting in nature. We first implement a strong MORL baseline called PROMPTED-MORL where there are multiple reward signals for each of the objectives (preferences) given via prompts during RL training. Next, we propose PERSONALIZED SOUPS, a method that circumvents simultaneously optimizing multiple preferences by first optimizing multiple policy models each with distinct preferences with Proximal Policy Optimization (PPO) and merging the parameters of the policy models whose preferences we want to composite together on the fly during inference. This modular approach significantly reduces the computational complexity from exponential to linear in relation to the total number of unique preferences. Furthermore, since PERSONALIZED SOUPS does not have to be trained in a multitask fashion, it does not require re-training the underlying policy every time a novel preference (objective) is added. We empirically show that by transforming the problem of aligning LLMs to human preferences into a MORL problem, we are able to have personalized alignment that provides a deeper level of adaptation to individual users that supervised fine-tuning, RLHF, and prompting cannot attain. We also emphasize the modularity of PERSONALIZED SOUPS by performing experiments in a scenario where the user additionally writes novel preferences that they want to integrate with existing preferences. We show that in this scenario, PERSONALIZED SOUPS still performs competitively to PROMPTED-MORL while being exponentially more efficient through parameter merging. [1] Code: https://github.com/joeljang/RLPHF

Personalized Soups: LLM Alignment Via Parameter Merging - Abstract & Introduction

This interview series celebrates all the phenomenal . Well-Deserved, Champion. HackerNoon sits down with Jett Prescott of PennyFly Entertainment: Startups of the Year Winners [ ] https://hackernoon.com/startups-of-the-year-2023-winners-north-america As the founder of PennyFly Entertainment, what does it mean for you to win this title? I am a long-time tech nerd and reader of HackerNoon, so it truly means a lot. Before PennyFly was even an idea, I graduated from Penn State with a bachelor’s degree in Cyber Security and was certified to work with the NSA. I fully expected my life would be immersed in all things hacking and counter-hacking, but shortly after graduating, what began as a hobby, writing and performing music turned into a serious career pursuit. Following label offers after moving to LA, I made the decision to stay independent while networking within the industry to build a team around me. This undertaking became much bigger than myself, snowballing into helping more and more artists, and today, we’ve worked with over 170 different artists at all career stages. As PennyFly grew, I began to wonder if I would ever utilize any of that education or if it was an unfortunate waste of time and money…thankfully, it wasn’t. Everything came full circle as blockchain and AI-driven solutions continued to dominate multiple major industries. My understanding of these technologies allowed PennyFly to integrate them into our services and workflows and ultimately expanded what we do into a new company division, as we have now catered to over a dozen web3-based projects. Big shoutout to our CTO, Dr. Harry Malinski, and CMO, Skye Ruedas, for spearheading these efforts! Tell us how your startup is changing the world. I often tell people we’re not exactly “reinventing the wheel” when it comes to artist services. We have mentored under some of the best marketing professionals in the industry and are simply offering the same major-label caliber services you would typically get from signing a larger label deal - Having been around for nearly seven years, we continue to be one of the only true label services agencies in the world that can replicate these niche services under one roof (and without the 360-style strings attached). In the past two years, we have seen increasingly heavy interest from artists outside of the US in need of our services, as they are unable to find what we do anywhere else. what truly makes us different is that we don’t gate-keep these services behind contracts that take ownership, rights, and freedoms away from artists. What do you love about your team, and why are you the ones to solve this problem? I feel incredibly blessed to be working alongside such passionate and driven individuals - but most of all, we’ve developed strong friendships over the years and consider each other family. As our artists know well, we are far from the stereotypical “corporate” vibes you see with most companies, especially other marketing agencies and labels. This goes a very long way when it comes to cultivating the right atmosphere for artists to thrive. While we are very results-driven and don’t take shortcuts, we’re also all close friends who enjoy hanging out at the end of a long day…from gaming to watching sports or streaming the latest South Park, we always make time to have some fun. Speaking of family, my wife Raye has been such an integral part of PennyFly becoming what it is today; she is an absolute powerhouse when it comes to dialing in the right branding for our artists, from web development to digital design. In contrast, Harry Malinski’s wife, Melody, has been a huge asset on the administrative side and is growing into a larger creative-driven role to complement our evolving needs in-house. What sets you apart from the competition? It really comes down to our unique combination of proven marketing techniques with real accessibility. We don’t do 360 deals or require artists to sign unfavorable contracts in order to get the same teams and services they would otherwise get from well-known labels. It is also entirely up to the artist whether or not they’d like to publicly mention involvement with us - some artists opt for a “silent partnership” where we support them behind the scenes without the portrayal of having been officially “signed”. There are pros and cons to both approaches, as plenty of artists also prefer the opposite, which is why we ultimately leave it up to them. The only thing that really matters to us is ensuring our artists are comfortable, motivated, and inspired to succeed. Lastly, we seem to be the only company in our niche that showcases a long-form compilation of testimonials featuring a wide variety of artists we’ve worked with, all on camera talking about their experience working with us. We treat our artists like family, which leads not only to success but also to having a lot of fun along the way. How do you or your company intend to embrace the responsibility of this title in 2024? What goals are you looking forward to accomplishing? We are simply going to continue doing what we do best: offering great teams to great artists. The traction we gain with our clients continues to speak for itself, and we are growing every week. A major focus for us in 2024 will be scaling up everything we do. This means building out our departments with additional teams to handle more artists without compromising our attention to detail, ability to deliver, and overall results. As our Director of Project Management, Carter Perran has been invaluable in leading successful recruitment efforts and keeping everyone on track. Are there any artists you are particularly excited about? Absolutely. One of our biggest passions is working with artists who have incredible talent and potential but need the right exposure to build a truly engaged fanbase. We are expecting big things from Tanya Mezh, Kiraly, J. Lauryn, Amelia Milo, CasPark, and Laura Warshauer, whom we’ve all thoroughly enjoyed working with. There are also some exciting new partnerships in the works as we speak! 2023 has been another crazy year, especially in tech with the Generative AI takeover - is this a trend you are concerned about? This really is an interesting topic that has many nuances to it. On one hand, we’ve seen an incredible improvement with some of our services that are now supported by AI. From increased efficiency to better results, it’s hard to argue with the advantages these AI tools provide when utilized properly. On the other hand, we can’t forget the value brought by real artists, musicians, and writers. There is a human element that will always transcend machines attempting to reproduce that human feel. Currently, getting the most out of these AI solutions still requires a skilled, intelligent professional to gather some of what the AI produces in order to shape the final result into something practical and presentable. Any words of wisdom you’d like to share with us? Whether you are an artist in the early stages of your career or already established, remember that the #1 thing to always focus on is consistency. Staying power is everything. Artists we work with who find the most success in the quickest timeframes remain incredibly consistent in their music release schedules and social media presence. Even if you have an entire team behind you, work ethic is everything - this cannot be overstated enough. If you embody these values throughout your craft and your content and remain true to your authentic self, you can accomplish anything you set your mind to. Overnight success is years in the making as they say, so don’t get discouraged if things don’t take off as quickly as you expect them to. Stay the course and keep going. The ones who win are the ones who simply refuse to give up. Startups of The Year is HackerNoon’s Flagship community-driven event celebrating startups that survived and thrived in 2023. 30,000 startups across 4200+ cities and six continents participated this year to be crowned the best startup in their city. See our global winners announcement . here Companies that blog more get 67% more leads - Jumpstart your company’s blogging journey by purchasing our . Business Blogging Package

Stories, News & Mentions about CASPARAI

Casparai’s Stories on HackerNoon