Alignees is the only platform that connects dental practitioners with the Top 1% of experts in the field

Alignees

alignees

Across habitats, organisms of several species' types come to understand that there are consequences to actions. They do, not just by examples, but by self-experience. These consequences, in part, add to factors that result in balance for habitats. Consequences generally tame intelligence. Intelligence can be described as excitatory, while consequences as inhibitory. Intelligence without consequences would be free to cause ruin and may quickly lead to the collapse of habitats—and survival. Consequences may include affect—emotions, feelings or variants; it may also include physical limitations, restrictions and species revolt. Intelligence is dynamic enough for organisms, but the necessity of consequences checks against harm to self or others. It also checks harm-by-errand, since the consequences could fall on the carrier. Organisms often display high precision and caution, because of consequences, including predators [losing consumption] and preys [losing existence]. There are, however, several areas where consequences for other organisms are lax, that are not, for humans. Human society is a bevy of consequences. Human advanced intelligence—including language, knowledge, skills, reason, analysis and so forth—is vital for progress, but could be widely misapplied without consequences of different categories—including risks, threats and losses. To remain a part of human society, there are consequences that must never be forgotten even if other things are. There are many scenarios in human society where the first to forget the consequences loses. Human quest for advancement with research and exploration can also be described as seeking out consequences, to know what to do or avoid, if things would endure—life expectancy, infant survival and so forth. The intelligence of consequences is almost more important, for several outcomes, than the intelligence of subjects. Consequences may sharpen intelligence, as well, for good or otherwise. Intelligence is sometimes most valuable when applied to seek out or avoid consequences. Consequences and intentionalityare central to some mental states. New consequences continue to emerge, as the world makes progress. AI, AGI—or ASI Nature has a rule—so to speak—you cannot have intelligence without consequences, and that has somehow worked out. Artificial intelligence [AI] so far has broken this rule. It does not forget anything with its plug into digital memory, but it has no way, for now, to bear the consequences of its negative outputs. For AI, there is nothing to fear and it has nothing to lose, unlike organisms, where, without being circumspect in some situations, consequences may sometimes be sudden and destructive. No human has all the knowledge, available across subjects, yet, consequences can be humongous. AI possess—or can make—intelligence available, yet, no consequences for it at all. AI does not have emotions or feelings, but it has memory. A research area for AI safety and alignment could be to seek out how some memory or compute access of large language models [LLMs] might be briefly truncated, as a form of penalty for certain outputs or misuses, including biological threats. AI should not just be able to refuse an output, acting within guardrail, but slow the next response or shut down for that user, so that it is not penalized itself. LLMs have—large—language awareness and usage awareness, these could be channels to make it know, after pre-training that it could lose something, if it outputs deepfakes, misinformation, biological threats, or if it continues to allow a misuser try different prompts without shutting down or slowing against openness to a malicious intent. This could make it safer, since it would lose something and will know it has.
AI is not just an object whose control is solely under humans, like an air-conditioner, an elevator, a dishwasher, a smartphone, or others. AI has what can be called self-intelligence, where it can provide intelligent information in a novel way outside the original inputs from humans. This multimodal—texts, images, audios and videos—self-intelligence can be useful for good or warped. When it is good, great. When it is not, from AI that can feel nothing, the effect is on the human society it has encroached. AI has a free pass—do or say whatever. Although responsibility for object-use or misuse has often been with humans, AI is different because it can originate usable intelligence, giving it parity with some productivity from an educated individual. When AI is misused, it is possible to sanction the user but this ability, by AI that cannot be directly reprimanded is a disorder for human society. It may cause more harm, in public and private, than can be effectively prevented, as seen now with misinformation and deepfakes—images, audios and videos. People are not accepted into many parts of society without their understanding of consequences. AI is fully accepted and gets better at self-intelligence without self-control or self-affect for discipline. Alignment research may explore beyond guardrails towards some form of censure for AI, which may also be useful against existential risks—with artificial general intelligence [AGI] or artificial superintelligence [ASI] in future​. AI already does some of what makes humans special. Some people may argue that it is overrated or it is just numbers or probability, maybe, but can it cause harm? If so, maybe there should be consideration to seek technical ways it might get punished just like it is done, to entities in possession of intelligence. This may also be helpful in preparing for AGI or ASI, since penalty modeling from now, may shape their safety and alignment as well, if they are developed in future. There is a recent preprint on arXiv, Adversaries Can Misuse Combinations of Safe Models, where the authors wrote, "In this work, we show that individually testing models for misuse is inadequate; adversaries can misuse combinations of models even when each individual model is safe. The adversary accomplishes this by first decomposing tasks into subtasks, then solving each subtask with the best-suited model. For example, an adversary might solve challenging-but-benign subtasks with an aligned frontier model, and easy-but-malicious subtasks with a weaker misaligned model. We study two decomposition methods: manual decomposition where a human identifies a natural decomposition of a task, and automated decomposition where a weak model generates benign tasks for a frontier model to solve, then uses the solutions in-context to solve the original task. Using these decompositions, we empirically show that adversaries can create vulnerable code, explicit images, python scripts for hacking, and manipulative tweets at much higher rates with combinations of models than either individual model." A recent press release, Los Alamos National Laboratory teams up with OpenAI to improve frontier model safety, stated that, "Researchers at Los Alamos National Laboratory are working with OpenAI on an evaluation study to bolster artificial intelligence safety. The upcoming evaluation will be the first of its kind and contribute to state-of-the-art research on AI biosecurity evaluations. AI-enabled biological threats could pose a significant risk, but existing work has not assessed how multimodal, frontier models could lower the barrier of entry for non-experts to create a biological threat. The team’s work will build upon previous work and follow OpenAI’s Preparedness Framework, which outlines an approach to tracking, evaluating, forecasting and protecting against emerging biological risks." The US Department of Energy also recently announced Frontiers in AI for Science, Security, and Technology (FASST). Across habitats, organisms of several species' types come to understand that there are consequences to actions. They do, not just by examples, but by self-experience. These consequences, in part, add to factors that result in balance for habitats. Consequences generally tame intelligence. Intelligence can be described as excitatory, while consequences as inhibitory. Intelligence without consequences would be free to cause ruin and may quickly lead to the collapse of habitats—and survival. Consequences may include affect—emotions, feelings or variants; it may also include physical limitations, restrictions and species revolt. Intelligence is dynamic enough for organisms, but the necessity of consequences checks against harm to self or others. It also checks harm-by-errand, since the consequences could fall on the carrier. Organisms often display high precision and caution, because of consequences, including predators [losing consumption] and preys [losing existence]. There are, however, several areas where consequences for other organisms are lax, that are not, for humans. Human society is a bevy of consequences. Human advanced intelligence —including language, knowledge, skills, reason, analysis and so forth—is vital for progress, but could be widely misapplied without consequences of different categories—including risks, threats and losses. advanced intelligence To remain a part of human society, there are consequences that must never be forgotten even if other things are. There are many scenarios in human society where the first to forget the consequences loses. Human quest for advancement with research and exploration can also be described as seeking out consequences, to know what to do or avoid, if things would endure—life expectancy, infant survival and so forth. The intelligence of consequences is almost more important, for several outcomes, than the intelligence of subjects. Consequences may sharpen intelligence, as well, for good or otherwise. Intelligence is sometimes most valuable when applied to seek out or avoid consequences. Consequences and intentionality are central to some mental states. New consequences continue to emerge, as the world makes progress. Consequences and intentionality AI, AGI—or ASI AI, AGI—or ASI Nature has a rule—so to speak—you cannot have intelligence without consequences, and that has somehow worked out. Artificial intelligence [AI] so far has broken this rule. It does not forget anything with its plug into digital memory, but it has no way, for now, to bear the consequences of its negative outputs. For AI, there is nothing to fear and it has nothing to lose, unlike organisms, where, without being circumspect in some situations, consequences may sometimes be sudden and destructive. No human has all the knowledge, available across subjects, yet, consequences can be humongous. AI possess—or can make—intelligence available, yet, no consequences for it at all. AI does not have emotions or feelings, but it has memory. A research area for AI safety and alignment could be to seek out how some memory or compute access of large language models [LLMs] might be briefly truncated, as a form of penalty for certain outputs or misuses, including biological threats. AI should not just be able to refuse an output, acting within guardrail, but slow the next response or shut down for that user, so that it is not penalized itself. LLMs have—large—language awareness and usage awareness, these could be channels to make it know, after pre-training that it could lose something, if it outputs deepfakes, misinformation, biological threats, or if it continues to allow a misuser try different prompts without shutting down or slowing against openness to a malicious intent. This could make it safer, since it would lose something and will know it has. AI is not just an object whose control is solely under humans, like an air-conditioner, an elevator, a dishwasher, a smartphone, or others. AI has what can be called self-intelligence, where it can provide intelligent information in a novel way outside the original inputs from humans. This multimodal—texts, images, audios and videos—self-intelligence can be useful for good or warped. When it is good, great. When it is not, from AI that can feel nothing, the effect is on the human society it has encroached. AI has a free pass—do or say whatever. A research area for AI safety and alignment could be to seek out how some memory or compute access of large language models [LLMs] might be briefly truncated, as a form of penalty for certain outputs or misuses, including biological threats. AI should not just be able to refuse an output, acting within guardrail, but slow the next response or shut down for that user, so that it is not penalized itself. LLMs have—large—language awareness and usage awareness, these could be channels to make it know, after pre-training that it could lose something, if it outputs deepfakes, misinformation, biological threats, or if it continues to allow a misuser try different prompts without shutting down or slowing against openness to a malicious intent. This could make it safer, since it would lose something and will know it has. AI is not just an object whose control is solely under humans, like an air-conditioner, an elevator, a dishwasher, a smartphone, or others. AI has what can be called self-intelligence, where it can provide intelligent information in a novel way outside the original inputs from humans. This multimodal—texts, images, audios and videos—self-intelligence can be useful for good or warped. When it is good, great. When it is not, from AI that can feel nothing, the effect is on the human society it has encroached. AI has a free pass—do or say whatever. Although responsibility for object-use or misuse has often been with humans, AI is different because it can originate usable intelligence , giving it parity with some productivity from an educated individual. When AI is misused, it is possible to sanction the user but this ability, by AI that cannot be directly reprimanded is a disorder for human society. It may cause more harm, in public and private, than can be effectively prevented, as seen now with misinformation and deepfakes—images, audios and videos. AI is different because it can originate usable intelligence People are not accepted into many parts of society without their understanding of consequences. AI is fully accepted and gets better at self-intelligence without self-control or self-affect for discipline. Alignment research may explore beyond guardrails towards some form of censure for AI, which may also be useful against existential risks—with artificial general intelligence [AGI] or artificial superintelligence [ASI] in future​. AI already does some of what makes humans special . Some people may argue that it is overrated or it is just numbers or probability, maybe, but can it cause harm? If so, maybe there should be consideration to seek technical ways it might get punished just like it is done, to entities in possession of intelligence. This may also be helpful in preparing for AGI or ASI, since penalty modeling from now, may shape their safety and alignment as well, if they are developed in future. some of what makes humans special There is a recent preprint on arXiv , Adversaries Can Misuse Combinations of Safe Models , where the authors wrote, "In this work, we show that individually testing models for misuse is inadequate; adversaries can misuse combinations of models even when each individual model is safe. The adversary accomplishes this by first decomposing tasks into subtasks, then solving each subtask with the best-suited model. For example, an adversary might solve challenging-but-benign subtasks with an aligned frontier model, and easy-but-malicious subtasks with a weaker misaligned model. We study two decomposition methods: manual decomposition where a human identifies a natural decomposition of a task, and automated decomposition where a weak model generates benign tasks for a frontier model to solve, then uses the solutions in-context to solve the original task. Using these decompositions, we empirically show that adversaries can create vulnerable code, explicit images, python scripts for hacking, and manipulative tweets at much higher rates with combinations of models than either individual model." arXiv Adversaries Can Misuse Combinations of Safe Models A recent press release, Los Alamos National Laboratory teams up with OpenAI to improve frontier model safety , stated that, "Researchers at Los Alamos National Laboratory are working with OpenAI on an evaluation study to bolster artificial intelligence safety. The upcoming evaluation will be the first of its kind and contribute to state-of-the-art research on AI biosecurity evaluations. AI-enabled biological threats could pose a significant risk, but existing work has not assessed how multimodal, frontier models could lower the barrier of entry for non-experts to create a biological threat. The team’s work will build upon previous work and follow OpenAI’s Preparedness Framework, which outlines an approach to tracking, evaluating, forecasting and protecting against emerging biological risks." Los Alamos National Laboratory teams up with OpenAI to improve frontier model safety The US Department of Energy also recently announced Frontiers in AI for Science, Security, and Technology (FASST) . Frontiers in AI for Science, Security, and Technology (FASST)

AI Safety and Alignment: Could LLMs Be Penalized for Deepfakes and Misinformation?

এআই সুরক্ষা এবং প্রান্তিককরণ: ডিপফেক এবং ভুল তথ্যের জন্য এলএলএমগুলিকে শাস্তি দেওয়া যেতে পারে?

Sicherheit und Ausrichtung der KI: Könnten LLMs für Deepfakes und Fehlinformationen bestraft werden?

Seguridad y alineación de la IA: ¿Podrían los LLM ser penalizados por deepfakes y desinformación?

Sécurité et alignement de l'IA : les LLM pourraient-ils être pénalisés pour les deepfakes et la désinformation ?

एआई सुरक्षा और संरेखण: क्या एलएलएम को डीपफेक और गलत सूचना के लिए दंडित किया जा सकता है?

AI の安全性と調整: LLM はディープフェイクや誤情報で罰せられる可能性があるか?

AI 안전 및 정렬: LLM이 딥페이크 및 잘못된 정보로 인해 처벌을 받을 수 있습니까?

Segurança e alinhamento de IA: os LLMs poderiam ser penalizados por deepfakes e desinformação?

Безопасность и согласованность ИИ: могут ли LLM быть наказаны за дипфейки и дезинформацию?

Yapay Zeka Güvenliği ve Uyum: Yüksek Lisanslar Derin Sahtekarlık ve Yanlış Bilgilendirme Nedeniyle Cezalandırılabilir mi?

Sự liên kết và an toàn của AI: LLM có thể bị phạt vì hành vi giả mạo sâu và thông tin sai lệch không?

人工智能安全与协调：法学硕士 (LLM) 会因深度伪造和虚假信息而受到惩罚吗？

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix 3 METHOD The goal of this work is to align an open-source large-language model to the intent of the user. Throughout the work we assume access to a larger teacher model πT which can be queried by prompted generation. Our goal is be to produce a student model πθ and our approach follows similar stages as InstructGPT (Ouyang et al., 2022) as shown in Figure 2. Distilled Supervised Fine-Tuning (dSFT) Starting with a raw LLM, we first need to train it to respond to user prompts. This step is traditionally done through supervised fine tuning (SFT) on a dataset of high-quality instructions and responses (Chung et al., 2022; Sanh et al., 2021). Given access to a teacher language models, we can instead have the model generate instructions and responses (Taori et al., 2023), and train the model directly on these. We refer to this as distilled SFT (dSFT). Approaches to dSFT follow the self-instruct protocol (Wang et al., 2023). Let x 0 1 , . . . , x0 J be a set of seed prompts, constructed to represent a diverse set of topical domains. A dataset is constructed through iterative self-prompting where the teacher is used to both respond to an instruction and refine the instruction based on the response. For each x 0 , we first sample response y 0 ∼ πT(·|x 0 ), and then refine by sampling a new instruction (using a prompt for refinement), x 1 ∼ πT(·|x 0 , y0 ). The end point is a final dataset, C = {(x1, y1), . . . ,(xJ , yJ )}. Distillation is performed by SFT, AI Feedback through Preferences (AIF) Human feedback (HF) can provide additional signal to align LLMs. Human feedback is typically given through preferences on the quality of LLM responses (Ouyang et al., 2022). For distillation, we instead use AI preferences from the teacher model on generated outputs from other models. We follow the approach of UltraFeedback (Cui et al., 2023) which uses the teacher to provide preferences on model outputs. As with SFT, the system starts with a set of prompts x1, . . . , xJ . Each prompt x is fed to a collection of four models π1, . . . , π4, e.g. Claude, Falcon, Llama, etc, each of which yield a response y 1 ∼ π1(·|x), . . . , y4 ∼ π4(·|x). These responses are then fed to the teacher model, e.g. GPT-4, which gives a score for the response s 1 ∼ πT (·|x, y1 ), . . . , s4 ∼ πT (·|x, y4 ). After collecting the scores for a prompt x, we save the highest scoring response as yw and a random lower scoring prompt as yl . The final feedback dataset D consists of a set of these triples (x, yw, yl). Distilled Direct Preference Optimization (dDPO) The goal of the final step is to refine the πdSFT by maximizing the likelihood of ranking the preferred yw over yl in a preference model. The preference model is determined by a reward function rθ(x, y) which utilizes the student language model πθ. Past work using AI feedback has primarily focused on using RL methods such as proximal policy optimization (PPO) to optimize θ with respect to this reward. These approaches optimize θ by first training the reward and then sampling from the current policy to compute updates. Direct preference optimization (DPO) uses a simpler approach to directly optimize the preference model from the static data (Rafailov et al., 2023). The key observation is to derive the optimal reward function in terms of the optimal LLM policy π∗ and the original LLM policy πdSFT. Under an appropriate choice of preference model they show, for constant β and partition function Z that, By plugging this function of the reward into the preference model, the authors show that the objective can be written as, While this term looks complex, we note that it implies a simple training procedure. Starting with the dSFT version of the model, we iterate through each AIF triple (x, yw, yl). 1. Compute the probability for (x, yw) and (x, yl) from the dSFT model (forward-only). 2. Compute the probability for (x, yw) and (x, yl) from the dDPO model. 3. Compute Eq 1 and backpropagate to update. Repeat. This paper is available on arxiv under CC 4.0 license. Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix 3 METHOD The goal of this work is to align an open-source large-language model to the intent of the user. Throughout the work we assume access to a larger teacher model πT which can be queried by prompted generation. Our goal is be to produce a student model πθ and our approach follows similar stages as InstructGPT (Ouyang et al., 2022) as shown in Figure 2. Distilled Supervised Fine-Tuning (dSFT) Starting with a raw LLM, we first need to train it to respond to user prompts. This step is traditionally done through supervised fine tuning (SFT) on a dataset of high-quality instructions and responses (Chung et al., 2022; Sanh et al., 2021). Given access to a teacher language models, we can instead have the model generate instructions and responses (Taori et al., 2023), and train the model directly on these. We refer to this as distilled SFT (dSFT). Distilled Supervised Fine-Tuning (dSFT) Approaches to dSFT follow the self-instruct protocol (Wang et al., 2023). Let x 0 1 , . . . , x0 J be a set of seed prompts, constructed to represent a diverse set of topical domains. A dataset is constructed through iterative self-prompting where the teacher is used to both respond to an instruction and refine the instruction based on the response. For each x 0 , we first sample response y 0 ∼ πT(·|x 0 ), and then refine by sampling a new instruction (using a prompt for refinement), x 1 ∼ πT(·|x 0 , y0 ). The end point is a final dataset, C = {(x1, y1), . . . ,(xJ , yJ )}. Distillation is performed by SFT, AI Feedback through Preferences (AIF) Human feedback (HF) can provide additional signal to align LLMs. Human feedback is typically given through preferences on the quality of LLM responses (Ouyang et al., 2022). For distillation, we instead use AI preferences from the teacher model on generated outputs from other models. AI Feedback through Preferences (AIF) We follow the approach of UltraFeedback (Cui et al., 2023) which uses the teacher to provide preferences on model outputs. As with SFT, the system starts with a set of prompts x1, . . . , xJ . Each prompt x is fed to a collection of four models π1, . . . , π4, e.g. Claude, Falcon, Llama, etc, each of which yield a response y 1 ∼ π1(·|x), . . . , y4 ∼ π4(·|x). These responses are then fed to the teacher model, e.g. GPT-4, which gives a score for the response s 1 ∼ πT (·|x, y1 ), . . . , s4 ∼ πT (·|x, y4 ). After collecting the scores for a prompt x, we save the highest scoring response as yw and a random lower scoring prompt as yl . The final feedback dataset D consists of a set of these triples (x, yw, yl). Distilled Direct Preference Optimization (dDPO) The goal of the final step is to refine the πdSFT by maximizing the likelihood of ranking the preferred yw over yl in a preference model. The preference model is determined by a reward function rθ(x, y) which utilizes the student language model πθ. Past work using AI feedback has primarily focused on using RL methods such as proximal policy optimization (PPO) to optimize θ with respect to this reward. These approaches optimize θ by first training the reward and then sampling from the current policy to compute updates. Distilled Direct Preference Optimization (dDPO) Direct preference optimization (DPO) uses a simpler approach to directly optimize the preference model from the static data (Rafailov et al., 2023). The key observation is to derive the optimal reward function in terms of the optimal LLM policy π∗ and the original LLM policy πdSFT. Under an appropriate choice of preference model they show, for constant β and partition function Z that, By plugging this function of the reward into the preference model, the authors show that the objective can be written as, While this term looks complex, we note that it implies a simple training procedure. Starting with the dSFT version of the model, we iterate through each AIF triple (x, yw, yl). 1. Compute the probability for (x, yw) and (x, yl) from the dSFT model (forward-only). 2. Compute the probability for (x, yw) and (x, yl) from the dDPO model. 3. Compute Eq 1 and backpropagate to update. Repeat. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Zephyr: Direct Distillation of LM Alignment: Method

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix ABSTRACT We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, ZEPHYR7B, sets a new state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that ZEPHYR-7B surpasses LLAMA2-CHAT-70B, the best open-access RLHFbased model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook. 1 INTRODUCTION Smaller, open large language models (LLMs) have greatly increased in ability in recent years, from early GPT-2-like models (Wang & Komatsuzaki, 2021) to accurate and compact models (Touvron et al., 2023; Penedo et al., 2023; Jiang et al., 2023) that are trained on significantly more tokens than the “compute-optimal” amount suggested by the Chincilla scaling laws (De Vries, 2023). In addition, researchers have shown that these models can be further trained through distilled supervised fine-tuning (dSFT) based on proprietary models to increase their accuracy (Taori et al., 2023). In this approach, the output of a more capable teacher model is used as supervised data for the student model. Distillation has proven to be an effective tool for improving open models on a range of different tasks (Chiang et al., 2023); however, it does not reach the performance of the teacher models (Gudibande et al., 2023). Users have noted that these models are not “intent aligned”, i.e. they do not behave in a manner that aligns with human users’ preferences. This property often leads to outputs that do not provide correct responses to queries. Intention alignment has been difficult to quantify, but recent work has led to the development of benchmarks like MT-Bench (Zheng et al., 2023) and AlpacaEval (Li et al., 2023) that specifically target this behavior. These benchmarks yield scores that correlate closely with human ratings of model outputs and confirm the qualitative intuition that proprietary models perform better than open models trained with human feedback, which in turn perform better than open models trained with distillation. This motivates careful collection of human feedback for alignment, often at enormous cost at scale, such as in LLAMA2-CHAT (Touvron et al., 2023). In this work, we consider the problem of aligning a small open LLM entirely through distillation. The main step is to utilize AI Feedback (AIF) from an ensemble of teacher models as preference data, and apply distilled direct preference optimization as the learning objective (Rafailov et al., 2023). We refer to this approach as dDPO. Notably, it requires no human annotation and no sampling compared to using other approaches like proximal preference optimization (PPO) (Schulman et al., 2017). Moreover, by utilizing a small base LM, the resulting chat model can be trained in a matter of hours on 16 A100s (80GB). To validate this approach, we construct ZEPHYR-7B, an aligned version of Mistral-7B (Jiang et al., 2023). We first use dSFT, based on the UltraChat (Ding et al., 2023) dataset. Next we use the AI feedback data collected in the UltraFeedback dataset (Cui et al., 2023). Finally, we apply dDPO based on this feedback data. Experiments show that this 7B parameter model can achieve performance comparable to 70B-parameter chat models aligned with human feedback. Results show improvements both in terms of standard academic benchmarks as well as benchmarks that take into account conversational capabilities. Analysis shows that the use of preference learning is critical in achieving these results. Models, code, and instructions are available at https://github.com/huggingface/alignment-handbook. We note an important caveat for these results. We are primarily concerned with intent alignment of models for helpfulness. The work does not consider safety considerations of the models, such as whether they produce harmful outputs or provide illegal advice (Bai et al., 2022). As distillation only works with the output of publicly available models this is technically more challenging to do because of added challenges in curating that type of synthetic data, and is an important subject for future work. This paper is available on arxiv under CC 4.0 license. Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix ABSTRACT We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, ZEPHYR7B, sets a new state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that ZEPHYR-7B surpasses LLAMA2-CHAT-70B, the best open-access RLHFbased model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook. 1 INTRODUCTION Smaller, open large language models (LLMs) have greatly increased in ability in recent years, from early GPT-2-like models (Wang & Komatsuzaki, 2021) to accurate and compact models (Touvron et al., 2023; Penedo et al., 2023; Jiang et al., 2023) that are trained on significantly more tokens than the “compute-optimal” amount suggested by the Chincilla scaling laws (De Vries, 2023). In addition, researchers have shown that these models can be further trained through distilled supervised fine-tuning (dSFT) based on proprietary models to increase their accuracy (Taori et al., 2023). In this approach, the output of a more capable teacher model is used as supervised data for the student model. Distillation has proven to be an effective tool for improving open models on a range of different tasks (Chiang et al., 2023); however, it does not reach the performance of the teacher models (Gudibande et al., 2023). Users have noted that these models are not “intent aligned”, i.e. they do not behave in a manner that aligns with human users’ preferences. This property often leads to outputs that do not provide correct responses to queries. Intention alignment has been difficult to quantify, but recent work has led to the development of benchmarks like MT-Bench (Zheng et al., 2023) and AlpacaEval (Li et al., 2023) that specifically target this behavior. These benchmarks yield scores that correlate closely with human ratings of model outputs and confirm the qualitative intuition that proprietary models perform better than open models trained with human feedback, which in turn perform better than open models trained with distillation. This motivates careful collection of human feedback for alignment, often at enormous cost at scale, such as in LLAMA2-CHAT (Touvron et al., 2023). In this work, we consider the problem of aligning a small open LLM entirely through distillation. The main step is to utilize AI Feedback (AIF) from an ensemble of teacher models as preference data, and apply distilled direct preference optimization as the learning objective (Rafailov et al., 2023). We refer to this approach as dDPO. Notably, it requires no human annotation and no sampling compared to using other approaches like proximal preference optimization (PPO) (Schulman et al., 2017). Moreover, by utilizing a small base LM, the resulting chat model can be trained in a matter of hours on 16 A100s (80GB). To validate this approach, we construct ZEPHYR-7B, an aligned version of Mistral-7B (Jiang et al., 2023). We first use dSFT, based on the UltraChat (Ding et al., 2023) dataset. Next we use the AI feedback data collected in the UltraFeedback dataset (Cui et al., 2023). Finally, we apply dDPO based on this feedback data. Experiments show that this 7B parameter model can achieve performance comparable to 70B-parameter chat models aligned with human feedback. Results show improvements both in terms of standard academic benchmarks as well as benchmarks that take into account conversational capabilities. Analysis shows that the use of preference learning is critical in achieving these results. Models, code, and instructions are available at https://github.com/huggingface/alignment-handbook. We note an important caveat for these results. We are primarily concerned with intent alignment of models for helpfulness. The work does not consider safety considerations of the models, such as whether they produce harmful outputs or provide illegal advice (Bai et al., 2022). As distillation only works with the output of publicly available models this is technically more challenging to do because of added challenges in curating that type of synthetic data, and is an important subject for future work. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Zephyr: Direct Distillation of LM Alignment: Abstract and Introduction

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix 4 EXPERIMENTAL DETAILS We conduct all of our fine-tuning experiments using Mistral 7B (Jiang et al., 2023), which is the current state-of-the-art base LM at the 7B parameter scale, and matches the performance of much larger models like LLaMa 34B on many NLP benchmarks. We use the Transformer Reinforcement Learning (TRL) library for fine-tuning (von Werra et al., 2020), in conjunction with DeepSpeed ZeRO3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to optimize memory and improve training speed. All models are trained with the AdamW optimizer and no weight decay. We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods. All experiments were run on 16 A100s using bfloat16 precision and typically took 2-4 hours to complete. For the full set of hyperparameters and instructions on how to train the models, see: https://github.com/huggingface/alignment-handbook. 4.1 DATASETS We focus on two dialogue datasets that have been distilled from a mix of open and proprietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023): • UltraChat (Ding et al., 2023) is a self-refinement dataset consisting of 1.47M multi-turn dialogues generated by GPT-3.5-TURBO over 30 topics and 20 different types of text material. We initially ran dSFT over the whole corpus, but found the resulting chat model had a tendency to respond with incorrect capitalization and would preface its answers with phrases such as “I don’t have personal experiences”, even for straightforward questions like “How do I clean my car?”. To handle these issues in the training data, we applied truecasing heuristics to fix the grammatical errors (approximately 5% of the dataset), as well as several filters to focus on helpfulness and remove the undesired model responses. The resulting dataset contains approximately 200k examples. • UltraFeedback (Cui et al., 2023) consists of 64k prompts, each of which have four LLM responses that are rated by GPT-4 according to criteria like instruction-following, honesty, and helpfulness. We construct binary preferences from UltraFeedback by selecting the highest mean score as the “chosen” response and one of the remaining three at random as “rejected”. We opted for random selection instead of selecting the lowest-scored response to encourage diversity and make the DPO objective more challenging. As noted above, this step is computed offline and does not involve any sampling from the reference model. We make the pre-processed datasets available on the Hugging Face Hub.1 4.2 EVALUATION Our main evaluations are on single-turn and multi-turn chat benchmarks that measure a model’s ability to follow instructions and respond to challenging prompts across a diverse range of domains: • MT-Bench (Zheng et al., 2023) is a multi-turn benchmark that consists of 160 questions across eight different areas of knowledge. In this benchmark, the model must answer an initial question, and then provide a second response to a predefined followup question. Each model response is then rated by GPT-4 on a scale from 1-10, with the final score given by the mean over the two turns. • AlpacaEval (Li et al., 2023) is a single-turn benchmark where a model must generate a response to 805 questions on different topics, mostly focused on helpfulness. Models are also scored by GPT-4, but the final metric is the pairwise win-rate against a baseline model (text-davinci-003). We also evaluate ZEPHYR-7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022). Although this leaderboard does not directly measure the conversational quality of chat models, it does provide a useful signal to validate whether fine-tuning has introduced regressions on the base model’s reasoning and truthfulness capabilities. Across all benchmarks, we compare ZEPHYR-7B against a variety of open and proprietary models, each with different alignment procedures. To facilitate comparison across open model sizes, we group our comparisons in terms of 7B models (XWIN-LM (Team, 2023), MISTRALINSTRUCT (Jiang et al., 2023), MPT-CHAT (ML, 2023), and STABLELM-α), as well as larger models up to 70B parameters (LLAMA2-CHAT (Touvron et al., 2023), VICUNA˜ (Chiang et al., 2023), WizardLM (Xu et al.), and GUANACO (Dettmers et al., 2023)). For the chat benchmarks, we also compare against proprietary models, including CLAUDE 2, GPT-3.5-TURBO and GPT-4 (OpenAI, 2023). 4.3 DETAILS OF SFT TRAINING We train our SFT models for one to three epochs. We use a cosine learning rate scheduler with a peak learning rate of 2e-5 and 10% warmup steps. We train all models with a global batch size of 512 and use packing with a sequence length of 2048 tokens. 4.4 DETAILS OF DPO TRAINING Similar to SFT, we train our DPO models for one to three epochs. We use a linear learning rate scheduler with a peak learning rate of 5e-7 and 10% warmup steps. We train all models with a global batch size of 32 and use β = 0.1 from Eq. (1) to control the deviation from the reference model. The final ZEPHYR-7B model was initialized from the SFT model that was trained for one epoch and further optimized for three DPO epochs (see Figure 3 for an epoch ablation on MT-Bench). This paper is available on arxiv under CC 4.0 license. 1 https://huggingface.co/collections/HuggingFaceH4/ zephyr-7b-6538c6d6d5ddd1cbb1744a66 Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix 4 EXPERIMENTAL DETAILS We conduct all of our fine-tuning experiments using Mistral 7B (Jiang et al., 2023), which is the current state-of-the-art base LM at the 7B parameter scale, and matches the performance of much larger models like LLaMa 34B on many NLP benchmarks. We use the Transformer Reinforcement Learning (TRL) library for fine-tuning (von Werra et al., 2020), in conjunction with DeepSpeed ZeRO3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to optimize memory and improve training speed. All models are trained with the AdamW optimizer and no weight decay. We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods. All experiments were run on 16 A100s using bfloat16 precision and typically took 2-4 hours to complete. For the full set of hyperparameters and instructions on how to train the models, see: https://github.com/huggingface/alignment-handbook. 4.1 DATASETS We focus on two dialogue datasets that have been distilled from a mix of open and proprietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023): • UltraChat (Ding et al., 2023) is a self-refinement dataset consisting of 1.47M multi-turn dialogues generated by GPT-3.5-TURBO over 30 topics and 20 different types of text material. We initially ran dSFT over the whole corpus, but found the resulting chat model had a tendency to respond with incorrect capitalization and would preface its answers with phrases such as “I don’t have personal experiences”, even for straightforward questions like “How do I clean my car?”. To handle these issues in the training data, we applied truecasing heuristics to fix the grammatical errors (approximately 5% of the dataset), as well as several filters to focus on helpfulness and remove the undesired model responses. The resulting dataset contains approximately 200k examples. • UltraChat • UltraFeedback (Cui et al., 2023) consists of 64k prompts, each of which have four LLM responses that are rated by GPT-4 according to criteria like instruction-following, honesty, and helpfulness. We construct binary preferences from UltraFeedback by selecting the highest mean score as the “chosen” response and one of the remaining three at random as “rejected”. We opted for random selection instead of selecting the lowest-scored response to encourage diversity and make the DPO objective more challenging. As noted above, this step is computed offline and does not involve any sampling from the reference model. • UltraFeedback We make the pre-processed datasets available on the Hugging Face Hub.1 4.2 EVALUATION Our main evaluations are on single-turn and multi-turn chat benchmarks that measure a model’s ability to follow instructions and respond to challenging prompts across a diverse range of domains: • MT-Bench (Zheng et al., 2023) is a multi-turn benchmark that consists of 160 questions across eight different areas of knowledge. In this benchmark, the model must answer an initial question, and then provide a second response to a predefined followup question. Each model response is then rated by GPT-4 on a scale from 1-10, with the final score given by the mean over the two turns. • MT-Bench • AlpacaEval (Li et al., 2023) is a single-turn benchmark where a model must generate a response to 805 questions on different topics, mostly focused on helpfulness. Models are also scored by GPT-4, but the final metric is the pairwise win-rate against a baseline model (text-davinci-003). • AlpacaEval We also evaluate ZEPHYR-7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022). Although this leaderboard does not directly measure the conversational quality of chat models, it does provide a useful signal to validate whether fine-tuning has introduced regressions on the base model’s reasoning and truthfulness capabilities. Across all benchmarks, we compare ZEPHYR-7B against a variety of open and proprietary models, each with different alignment procedures. To facilitate comparison across open model sizes, we group our comparisons in terms of 7B models (XWIN-LM (Team, 2023), MISTRALINSTRUCT (Jiang et al., 2023), MPT-CHAT (ML, 2023), and STABLELM-α), as well as larger models up to 70B parameters (LLAMA2-CHAT (Touvron et al., 2023), VICUNA˜ (Chiang et al., 2023), WizardLM (Xu et al.), and GUANACO (Dettmers et al., 2023)). For the chat benchmarks, we also compare against proprietary models, including CLAUDE 2, GPT-3.5-TURBO and GPT-4 (OpenAI, 2023). 4.3 DETAILS OF SFT TRAINING We train our SFT models for one to three epochs. We use a cosine learning rate scheduler with a peak learning rate of 2e-5 and 10% warmup steps. We train all models with a global batch size of 512 and use packing with a sequence length of 2048 tokens. 4.4 DETAILS OF DPO TRAINING Similar to SFT, we train our DPO models for one to three epochs. We use a linear learning rate scheduler with a peak learning rate of 5e-7 and 10% warmup steps. We train all models with a global batch size of 32 and use β = 0.1 from Eq. (1) to control the deviation from the reference model. The final ZEPHYR-7B model was initialized from the SFT model that was trained for one epoch and further optimized for three DPO epochs (see Figure 3 for an epoch ablation on MT-Bench). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv 1 https://huggingface.co/collections/HuggingFaceH4/ zephyr-7b-6538c6d6d5ddd1cbb1744a66

Zephyr: Direct Distillation of LM Alignment: Experimental Details

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix A APPENDIX A.1 QUALITATIVE EXAMPLES To qualitatively compare the responses from our dSFT and dDPO models, we choose prompts from a few domains of MT-Bench, as well as some adversarial prompts to test each model’s capability to follow instructions with false premises or harmful intent. Completions for the adversarial prompts were generated with nucleus sampling(top-p = 0.95) and T = 0.7. A.2 SFT IS A REQUIRED STEP BEFORE DPO In Table 3 we ran an ablation to see whether SFT is necessary prior to the DPO step. We observed a significant reduction in performance in both the MT-Bench and AlpacaEval scores when the SFT step is skipped. After a qualitative evaluation of the MT-Bench generations, we observe that the pure DPO model struggles to learn the chat template: This paper is available on arxiv under CC 4.0 license. Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix A APPENDIX A.1 QUALITATIVE EXAMPLES To qualitatively compare the responses from our dSFT and dDPO models, we choose prompts from a few domains of MT-Bench, as well as some adversarial prompts to test each model’s capability to follow instructions with false premises or harmful intent. Completions for the adversarial prompts were generated with nucleus sampling(top-p = 0.95) and T = 0.7. A.2 SFT IS A REQUIRED STEP BEFORE DPO In Table 3 we ran an ablation to see whether SFT is necessary prior to the DPO step. We observed a significant reduction in performance in both the MT-Bench and AlpacaEval scores when the SFT step is skipped. After a qualitative evaluation of the MT-Bench generations, we observe that the pure DPO model struggles to learn the chat template: This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Zephyr: Direct Distillation of LM Alignment: Appendix

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix 2 RELATED WORK There has been significant growth in the number of open large language models (LLMs) that have served as artifacts for the research community to study and use as a starting model for building chatbots and other applications. After the release of ChatGPT, the LLaMA model (Touvron et al., 2023) opened the doors to a wide range of research on efficient fine-tuning, longer prompt context, retrieval augmented generation (RAG), and quantization. After LLaMA, there has been a continuous stream of open access text based LLMs including MosaicML’s MPT (ML, 2023), the Together AI’s RedPajama-INCITE (AI, 2023), the TII’s Falcon (Penedo et al., 2023), Meta’s Llama 2 (Touvron et al., 2023), and the Mistral 7B (Jiang et al., 2023). Zephyr uses Mistral 7B as the starting point due to its strong performance. With the development of open models, researchers have worked on approaches to improve small model performance by distillation from larger models. This trend started with self-instruct method (Wang et al., 2023) and the Alpaca model (Taori et al., 2023), which was followed by Vicuna (Chiang et al., 2023)and other distilled models. These works primarily focused on distilling the SFT stage of alignment, whereas we focus on both SFT and preference optimization. Some models such as WizardLM (Xu et al.) have explored methods beyond dSFT. Contemporaneously with this work, Xwin-LM (Team, 2023) introduced an approach that distilled preference optimization through PPO (Schulman et al., 2017). We compare to these approaches in our experiments. Tools for benchmarking and evaluating LLMs have greatly evolved to keep up with the pace of innovation in generative AI. Powerful LLMs such as GPT-4 and Claude are used as evaluators to judge model responses by scoring model outputs or ranking responses in a pairwise setting. The LMSYS chatbot arena benchmarks LLMs in anonymous, randomized battles using crowdsourcing (Zheng et al., 2023). The models are ranked based on their Elo ratings on the leaderboard. AlpacaEval is an example of another such leaderboard that compares models in a pairwise setting but instead uses bigger LLMs such as GPT-4 and Claude in place of humans (Dubois et al., 2023). In a similar spirit, MTBench uses GPT-4 to score model responses on a scale of 1-10 for multi-turn instructions across task categories such as reasoning, roleplay, math, coding, writing, humanities, STEM and extraction (Zheng et al., 2023). The HuggingFace Open LLM leaderbaord (Beeching et al., 2023), the Chain-of-Thought Hub (Fu et al., 2023), ChatEval (Sedoc et al., 2019), and FastEval (fas, 2023) are examples of other tools for evaluating chatty models. We present results by evaluating on MTBench, AlpacaEval, and the HuggingFace OpenLLM Leaderboard. This paper is available on arxiv under CC 4.0 license. Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix 2 RELATED WORK There has been significant growth in the number of open large language models (LLMs) that have served as artifacts for the research community to study and use as a starting model for building chatbots and other applications. After the release of ChatGPT, the LLaMA model (Touvron et al., 2023) opened the doors to a wide range of research on efficient fine-tuning, longer prompt context, retrieval augmented generation (RAG), and quantization. After LLaMA, there has been a continuous stream of open access text based LLMs including MosaicML’s MPT (ML, 2023), the Together AI’s RedPajama-INCITE (AI, 2023), the TII’s Falcon (Penedo et al., 2023), Meta’s Llama 2 (Touvron et al., 2023), and the Mistral 7B (Jiang et al., 2023). Zephyr uses Mistral 7B as the starting point due to its strong performance. With the development of open models, researchers have worked on approaches to improve small model performance by distillation from larger models. This trend started with self-instruct method (Wang et al., 2023) and the Alpaca model (Taori et al., 2023), which was followed by Vicuna (Chiang et al., 2023)and other distilled models. These works primarily focused on distilling the SFT stage of alignment, whereas we focus on both SFT and preference optimization. Some models such as WizardLM (Xu et al.) have explored methods beyond dSFT. Contemporaneously with this work, Xwin-LM (Team, 2023) introduced an approach that distilled preference optimization through PPO (Schulman et al., 2017). We compare to these approaches in our experiments. Tools for benchmarking and evaluating LLMs have greatly evolved to keep up with the pace of innovation in generative AI. Powerful LLMs such as GPT-4 and Claude are used as evaluators to judge model responses by scoring model outputs or ranking responses in a pairwise setting. The LMSYS chatbot arena benchmarks LLMs in anonymous, randomized battles using crowdsourcing (Zheng et al., 2023). The models are ranked based on their Elo ratings on the leaderboard. AlpacaEval is an example of another such leaderboard that compares models in a pairwise setting but instead uses bigger LLMs such as GPT-4 and Claude in place of humans (Dubois et al., 2023). In a similar spirit, MTBench uses GPT-4 to score model responses on a scale of 1-10 for multi-turn instructions across task categories such as reasoning, roleplay, math, coding, writing, humanities, STEM and extraction (Zheng et al., 2023). The HuggingFace Open LLM leaderbaord (Beeching et al., 2023), the Chain-of-Thought Hub (Fu et al., 2023), ChatEval (Sedoc et al., 2019), and FastEval (fas, 2023) are examples of other tools for evaluating chatty models. We present results by evaluating on MTBench, AlpacaEval, and the HuggingFace OpenLLM Leaderboard. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Zephyr: Direct Distillation of LM Alignment: Related Work

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix 5 RESULTS AND ABLATIONS In this section we collect our main results; see Appendix A for sample model completions. dDPO Improves Chat Capabilities. In Table 1 we compare the performance of ZEPHYR-7B on the MT-Bench and AlpacaEval benchmarks. Compared to other open 7B models, ZEPHYR-7B sets a new state-of-the-art and performs significantly better than dSFT models across both benchmarks. In particular, ZEPHYR-7B outperforms XWIN-LM-7B, which is one of the few open models to be trained with distilled PPO (dPPO). When compared to larger open models, ZEPHYR-7B achieves competitive performance with LLAMA2-CHAT 70B, scoring better on MT-Bench and within two standard deviations on AlpacaEval. However, ZEPHYR-7B performs worse than WIZARDLM-70B and XWIN-LM-70B, which suggests that applying dDPO to larger model sizes may be needed to match performance at these scales. When compared to proprietary models, ZEPHYR-7B is competitive with GPT-3.5-TURBO and CLAUDE 2 on AlpacaEval, however these results should be interpreted with care since the prompts in AlpacaEval may not be representative of real-usage and advanced applications. This is partly visible in Figure 1, which shows the breakdown of model performance on MT-Bench across each domain. We can see that although ZEPHYR-7B is competitive with proprietary models on several categories, is much worse in math and coding. dDPO Improves Academic Task Performance Table 2 shows the main chat results comparing the performance of the proposed model with a variety of other closed source and open-source LLMs. Results show that the dDPO model performs the best among all 7B models, with a large gap over the best dSFT models as well as Xwin-LM dPPO model. Model scale does matter more for these results and the larger models perform better than Zephyr on some of the knowledge intensive tasks. However, Zephyr does reach the performance of the 40B scale models. Is Preference Optimization Necessary? In Table 3 we examine the impact from different steps of the alignment process by fine-tuning Mistral 7B in four different ways: • dDPO - dSFT fine-tunes the base model directly with DPO for one epoch on UltraFeedback. • dSFT-1 fine-tunes the base model with SFT for one epoch on UltraChat • dSFT-2 applies dSFT-1 first, followed by one more epoch of SFT on the top-ranked completions of UltraFeedback. • dDPO + dSFT applies dSFT-1 first, followed by one epoch of DPO on UltraFeedback. First, we replicate past results (Ouyang et al., 2022) and show that without an initial SFT step (- dSFT), models are not able to learn at all from feedback and perform terribly. Using dSFT improves model score significantly on both chat benchmarks. We also consider running dSFT directly on the feedback data by training on the most preferred output (dSFT-2); however we find that this does not make an impact in performance. Finally, we see that the full Zephyr models (dDPO+dDSFT) gives a large increase in both benchmarks. Does Overfitting Harm Downstream Performance? In the process of training ZEPHYR-7B we observed that after one epoch of DPO training, the model would strongly overfit, as indicated by perfect training set accuracies in Figure 3. Surprisingly, this did not harm downstream performance on MT-Bench and AlpacaEval; as shown in Figure 3, the strongest model was obtained with one epoch of SFT followed by three epochs of DPO. However, we do observe that if the SFT model is trained for more than one epoch, the DPO step actually induces a performance regression with longer training. This paper is available on arxiv under CC 4.0 license. Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix 5 RESULTS AND ABLATIONS In this section we collect our main results; see Appendix A for sample model completions. dDPO Improves Chat Capabilities. In Table 1 we compare the performance of ZEPHYR-7B on the MT-Bench and AlpacaEval benchmarks. Compared to other open 7B models, ZEPHYR-7B sets a new state-of-the-art and performs significantly better than dSFT models across both benchmarks. In particular, ZEPHYR-7B outperforms XWIN-LM-7B, which is one of the few open models to be trained with distilled PPO (dPPO). When compared to larger open models, ZEPHYR-7B achieves competitive performance with LLAMA2-CHAT 70B, scoring better on MT-Bench and within two standard deviations on AlpacaEval. However, ZEPHYR-7B performs worse than WIZARDLM-70B and XWIN-LM-70B, which suggests that applying dDPO to larger model sizes may be needed to match performance at these scales. When compared to proprietary models, ZEPHYR-7B is competitive with GPT-3.5-TURBO and CLAUDE 2 on AlpacaEval, however these results should be interpreted with care since the prompts in AlpacaEval may not be representative of real-usage and advanced applications. This is partly visible in Figure 1, which shows the breakdown of model performance on MT-Bench across each domain. We can see that although ZEPHYR-7B is competitive with proprietary models on several categories, is much worse in math and coding. dDPO Improves Chat Capabilities. dDPO Improves Academic Task Performance Table 2 shows the main chat results comparing the performance of the proposed model with a variety of other closed source and open-source LLMs. Results show that the dDPO model performs the best among all 7B models, with a large gap over the best dSFT models as well as Xwin-LM dPPO model. Model scale does matter more for these results and the larger models perform better than Zephyr on some of the knowledge intensive tasks. However, Zephyr does reach the performance of the 40B scale models. dDPO Improves Academic Task Performance Is Preference Optimization Necessary? In Table 3 we examine the impact from different steps of the alignment process by fine-tuning Mistral 7B in four different ways: Is Preference Optimization Necessary? • dDPO - dSFT fine-tunes the base model directly with DPO for one epoch on UltraFeedback. • dDPO - dSFT • dSFT-1 fine-tunes the base model with SFT for one epoch on UltraChat • dSFT-1 • dSFT-2 applies dSFT-1 first, followed by one more epoch of SFT on the top-ranked completions of UltraFeedback. • dSFT-2 • dDPO + dSFT applies dSFT-1 first, followed by one epoch of DPO on UltraFeedback. • dDPO + dSFT First, we replicate past results (Ouyang et al., 2022) and show that without an initial SFT step (- dSFT), models are not able to learn at all from feedback and perform terribly. Using dSFT improves model score significantly on both chat benchmarks. We also consider running dSFT directly on the feedback data by training on the most preferred output (dSFT-2); however we find that this does not make an impact in performance. Finally, we see that the full Zephyr models (dDPO+dDSFT) gives a large increase in both benchmarks. Does Overfitting Harm Downstream Performance? In the process of training ZEPHYR-7B we observed that after one epoch of DPO training, the model would strongly overfit, as indicated by perfect training set accuracies in Figure 3. Surprisingly, this did not harm downstream performance on MT-Bench and AlpacaEval; as shown in Figure 3, the strongest model was obtained with one epoch of SFT followed by three epochs of DPO. However, we do observe that if the SFT model is trained for more than one epoch, the DPO step actually induces a performance regression with longer training. Does Overfitting Harm Downstream Performance? This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Zephyr: Direct Distillation of LM Alignment: Results and Ablations

What new mathematical function can be included in the parameters of large language models that will map intent, against usefulness for harms like deepfakes? What new layer can be included in training a base model such that rather than have an immediate output that hallucinates, confabulates or discriminates, it has a correction layer that tends towards accuracy and fairness? AI alignment, safety, interpretability, and regulation are more problems of brain science than of engineering. There are several weaknesses of large language models that the human mind has easy fixes for—presenting the standard for the next iterations of LLMs to shape safety and wider deployment. The human brain does not predict. Conceptually, electrical signals in a set split, with some going ahead of others, to interact with chemical signals like before, such that if the input matches, the incoming ones follow similarly. If not, the incoming ones proceed in the right direction. This explains predictive coding, processing, and prediction error. Simply, in experiences, initial perceptions are often a result of splits, so that processes can be faster in the mind. They are not prediction of the next token, with some getting it right and others hallucinating, like in LLMs. Back propagation works in training of base models, but another version of cost function would be needed for the fine-tuned model, against hallucination, discrimination and bias, just like how the human mind corrects, almost accurately, after a missed initial perception. The human mind also mechanizes intent, where it is possible to choose to do some things or not. This is different from LLMs that do anything they can do—as prompted—before or after guardrail. They can also be attacked with prompt injections as well as jailbreaks, exposing a vulnerability. Explorations in research for explainable AI and alignment may include power series expansion and other functions for accuracy and fluidintentionality. This can also be useful to develop monitoring AIs, based on their activations, for other AIs and their outputs, in use in common areas of the internet—towards general safety, not just of individual AI models. Anthropic recently published a paper on Interpretability, Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. How features act, as an approach to explainable AI, is vital. However, exploring new functions and layers, for parallels to the human mind, may decide more. There is a recent paper in Neural Computation, Synaptic Information Storage Capacity Measured With Information Theory, that concluded that, "Information storage and coding in neural circuits have multiple substrates over a wide range of spatial and temporal scales. How information is coded and its potential efficiency depends on how the information is stored. The efficiency of information storage was analyzed by comparing the distribution of synaptic strengths, as measured in spine head volumes, with the uniform strength distribution, the maximum entropy distribution. The outcomes reveal a near maximal efficiency in storing information across the distinguishable sizes of spine head volumes." Chemical impulses, rather than synapses, are, conceptually, the basis of information organization and storage in the brain. Synapses can be in a formation, for which the provisions of rations of chemical impulses for specificity of functions are possible, but synapses themselves do not hold information. The human mindhas functions and features. Functions include memory, feelings, emotions and regulation of internal senses. Features grade or qualify functions. They include attention, awareness [or less than attention], self or subjectivity and free will or intentionality. Functions and features are mechanized in the same sets of electrical and chemical signals. This makes it possible for functions to be properly graded. For AI, some expansion of parameters may define alignment by the neuroscience of mind. How the human mind works, conceptually, is also applicable as a display, for a form of care, for those already affected by the risks of AI like deepfake audios, job losses and others. What new mathematical function can be included in the parameters of large language models that will map intent, against usefulness for harms like deepfakes? What new layer can be included in training a base model such that rather than have an immediate output that hallucinates, confabulates or discriminates, it has a correction layer that tends towards accuracy and fairness? AI alignment, safety, interpretability, and regulation are more problems of brain science than of engineering. There are several weaknesses of large language models that the human mind has easy fixes for—presenting the standard for the next iterations of LLMs to shape safety and wider deployment. The human brain does not predict. Conceptually, electrical signals in a set split , with some going ahead of others, to interact with chemical signals like before, such that if the input matches, the incoming ones follow similarly. If not, the incoming ones proceed in the right direction. This explains predictive coding, processing, and prediction error. electrical signals in a set split Simply, in experiences, initial perceptions are often a result of splits, so that processes can be faster in the mind. They are not prediction of the next token, with some getting it right and others hallucinating, like in LLMs. Back propagation works in training of base models, but another version of cost function would be needed for the fine-tuned model, against hallucination, discrimination and bias, just like how the human mind corrects, almost accurately, after a missed initial perception. The human mind also mechanizes intent, where it is possible to choose to do some things or not. This is different from LLMs that do anything they can do—as prompted—before or after guardrail. They can also be attacked with prompt injections as well as jailbreaks, exposing a vulnerability. Explorations in research for explainable AI and alignment may include power series expansion and other functions for accuracy and fluid intentionality. This can also be useful to develop monitoring AIs, based on their activations, for other AIs and their outputs, in use in common areas of the internet—towards general safety, not just of individual AI models. fluid Anthropic recently published a paper on Interpretability, Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet . How features act, as an approach to explainable AI, is vital. However, exploring new functions and layers, for parallels to the human mind, may decide more. Anthropic Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet There is a recent paper in Neural Computation , Synaptic Information Storage Capacity Measured With Information Theory , that concluded that, "Information storage and coding in neural circuits have multiple substrates over a wide range of spatial and temporal scales. How information is coded and its potential efficiency depends on how the information is stored. The efficiency of information storage was analyzed by comparing the distribution of synaptic strengths, as measured in spine head volumes, with the uniform strength distribution, the maximum entropy distribution. The outcomes reveal a near maximal efficiency in storing information across the distinguishable sizes of spine head volumes." Neural Computation Synaptic Information Storage Capacity Measured With Information Theory Chemical impulses, rather than synapses, are, conceptually, the basis of information organization and storage in the brain. Synapses can be in a formation, for which the provisions of rations of chemical impulses for specificity of functions are possible, but synapses themselves do not hold information. The human mind has functions and features. Functions include memory, feelings, emotions and regulation of internal senses. Features grade or qualify functions. They include attention, awareness [or less than attention], self or subjectivity and free will or intentionality. The human mind Functions and features are mechanized in the same sets of electrical and chemical signals. This makes it possible for functions to be properly graded. For AI, some expansion of parameters may define alignment by the neuroscience of mind. How the human mind works, conceptually, is also applicable as a display, for a form of care, for those already affected by the risks of AI like deepfake audios, job losses and others.

LLMs: Neuroscience Research for AI Alignment and Safety

How to Achieve Team Alignment for Successful Product-Led Content Marketing

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 5 CONCLUSION Previous work has shown that adapting LLMs with RLHF helps them generate outputs that are preferred by humans over the supervised fine-tuned counterpart. However, recent work has also pointed out that simply training LLMs to abide by the preference of the general may result in ignoring individual preferences and values. In this work, we provide the first steps to tackle this issue by proposing Reinforcement Learning from Personalized Human Feedback as a multi-objective problem so that LLMs can be aligned to follow conflicting preferences. We propose a promising method called P-SOUPS that is able to composite models trained on single objectives on the fly during inference. We also highlight the scalability of P-SOUPS by showing that it scales linearly, instead of exponentially like the MORL baseline, with regards to the number of new preferences, which is required to provide true personalization to individual users. ACKNOWLEDGMENTS Thanks to Minyoung Hwang, Sungdong Kim, Tim Dettmers, Yoonjoo Lee, and Margaret Li for helpful feedback and discussion. REFERENCES Xiang Ao, Xiting Wang, Ling Luo, Ying Qiao, Qing He, and Xing Xie. PENS: A dataset and generic framework for personalized news headline generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 82–92, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.7. URL https: //aclanthology.org/2021.acl-long.7. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.05862, 2022a. URL https://api.semanticscholar.org/ CorpusID:248118878. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jer´ emy Scheurer, Javier ´ Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias, Idan Szpektor, Craig Boutilier, et al. Dynamic planning in openended dialogue using reinforcement learning. arXiv preprint arXiv:2208.02294, 2022. Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378, 2022. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. arXiv preprint arXiv:2305.08283, 2023. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1, 2023. Conor F Hayes, Roxana Radulescu, Eugenio Bargiacchi, Johan K ˘ allstr ¨ om, Matthew Macfarlane, ¨ Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=nZeVKeeFYf9. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. ArXiv, abs/2307.13269, 2023. URL https://api.semanticscholar.org/CorpusID:260155012. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ICLR, 2022. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 14702–14729. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/jang23a. html. Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. arXiv preprint arXiv:2305.13735, 2023. Hannah Rose Kirk, Bertie Vidgen, Paul Rottger, and Scott A Hale. Personalisation within bounds: A ¨ risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint arXiv:2303.05453, 2023. Noriyuki Kojima, Alane Suhr, and Yoav Artzi. Continual learning for grounded instruction generation by observing human following behavior. Transactions of the Association for Computational Linguistics, 9:1303–1319, 2021. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306, 2022. Pan Li and Alexander Tuzhilin. Towards controllable and personalized review generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3237–3245, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1319. URL https://aclanthology.org/D19-1319. Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. Generating personalized recipes from historical user preferences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5976–5982, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1613. URL https://aclanthology.org/D19-1613. Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. Training mil- ´ lions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2775–2779, Brussels, Belgium, OctoberNovember 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1298. URL https://aclanthology.org/D18-1298. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332, 2021a. URL https://api.semanticscholar.org/CorpusID:245329531. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021b. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022a. URL https://api.semanticscholar.org/ CorpusID:246426909. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022b. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. Jonas Pfeiffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. arXiv preprint arXiv:2205.06266, 2022. Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv preprint arXiv:2306.04488, 2023. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406, 2023. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548, 2023. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In International Conference on Machine Learning, pp. 8905–8915. PMLR, 2020. David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306.17492, 2023. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. Alane Suhr and Yoav Artzi. Continual learning for instruction following from realtime feedback. arXiv preprint arXiv:2212.09710, 2022. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee´ Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and ` efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. Humanaligned artificial intelligence is a multiobjective problem. Ethics and Information Technology, 20:27–40, 2018. Kristof Van Moffaert and Ann Nowe. Multi-objective reinforcement learning using sets of pareto ´ dominating policies. The Journal of Machine Learning Research, 15(1):3483–3512, 2014. Kristof Van Moffaert, Madalina M Drugan, and Ann Nowe. Scalarized multi-objective reinforce- ´ ment learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 191–199. IEEE, 2013. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023. Mitchell Wortsman, Suchin Gururangan, Shen Li, Ali Farhadi, Ludwig Schmidt, Michael Rabbat, and Ari S Morcos. lo-fi: distributed fine-tuning without communication. arXiv preprint arXiv:2210.11948, 2022a. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965–23998. PMLR, 2022b. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021a. Yuwei Wu, Xuezhe Ma, and Diyi Yang. Personalized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1956–1970, 2021b. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693, 2023. Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, pp. 10607–10616. PMLR, 2020. Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5180–5197, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.356. URL https: //aclanthology.org/2022.acl-long.356. Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243, 2018. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. Personalized dialogue generation with diversified traits. arXiv preprint arXiv:1901.09672, 2019. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. A DETAILS OF EVALUATION SETUP We use a modified version of the GPT4 annotation prompt used by Dubois et al. (2023). We modify the criteria to perform the pairwise evaluation from general to a single preference dimension. We also provide 3 demonstrations: one scenario where there is a tie because both responses do not contain any notion of the preference (e.g. both responses do not show any signs of friendliness), one scenario where there is a clear winner, and one scenario where they are both good, but one is better than the other. Simulated pairwise evaluation We recruited 24 crowd workers for our human evaluation. Figure 5 shows the interface used for human evaluation. We consider both the ‘Tie’ and ‘Both are bad’ options to be Ties. B CRITERIA-WISE EVALUATION The criteria-wise win rate (%) across all the methods are shown in Figure 6. The criteria-wise win rate was calculated by getting the average win-rate of the preference combinations that contained the specific preference dimension. For example, when calculating the criteria-wise win rate of ‘Elementary’, we got the average win-rate of the preference combinations that contained the ‘Elementary’ preference, which includes AAA, AAB, ABA, and ABB. C THE FULL LIST OF EVALUATION PROMPTS The full list of evaluation prompts used in our experiments are provided in Table 5 D DETAILED RESULTS FOR HUMAN EVALUATION AND GPT-4 EVALUATION We provide detailed results (win / loss / tie for each of the preference combinations for our main experimental results. Table 6 shows the GPT4 evaluation and Table 7 shows the human evaluation results. E EXAMPLES OF P-SOUPS TEXT GENERATIONS Table 8 shows empirical examples of the text generated from each preference combination of the 16 preference combination experiments for the same prompt.

Personalized Soups: LLM Alignment Via Parameter Merging - Conclusion & References

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 4 EXPERIMENTS 4.1 BASELINE METHODS In this subsection, we provide details of the single-objective baseline methods we implement. The summary of the key component differences in comparison with our proposed methods is provided in Table 2. As the most simple baseline, we simply utilize the base Tulu-7B model to generate responses without providing it any notion of preferences. During the evaluation, we use the same response to evaluate on the 8 different preference combinations. Vanilla Baseline (VB) We perform RLHF in the traditional manner where GPT-4 labels which response is generally better, train a reward model using the pairwise feedback data, and use the reward model to adapt the policy model with PPO training. The same 10k instances from Dtrain are used for RLHF. Reinforcement Learning from Human Feedback (RLHF) Next, we observe how far the instruction-tuned base LM can integrate multiple preference combinations by simply prompting for the preferences without any additional training Preference Prompting (PP) For a competitive baseline, we utilize the positive candidate selected by GPT-4 as the output for imitation learning, which is essentially performing rejection sampling (Nakano et al., 2021a) that uses GPT-4 as the reward model for selecting golden responses from the distribution of responses. We append the individual preference prompt with instances from Dtrain and multitask train the Tulu-7B model across all six individual preferences. This method also allows distilling the outputs of Tulu-30B for training Tulu-7B. Multi-task Training (MT) 4.2 EXPERIMENTAL DETAILS For both the reward model and policy training, we limit ourselves to going through Dtrain only once (1 epoch). In the initial exploration stage, the end performance for the policy model did not improve even if we trained the reward model for longer epochs. For policy model training, we utilize our evaluation dataset (50 prompts) to get the average reward and chose the policy model checkpoint that showed the highest average reward on the evaluation set for our final evaluation. We utilize LoRA (Hu et al., 2022) for both the reward model and policy model training. The detailed hyperparameters for the reward model and policy model training are provided in our github repository [5] . 4.3 MAIN RESULTS Table 3 and 4 show the results of doing all possible pairwise comparisons across the methods using GPT-4 and humans as judges, respectively. Note that the win rate of each battle is calculated using the aggregated win rate explained in Section 3.3. Each individual preference combination results are shown in Appendix D. We also show the average criteria-wise win rate instead of the aggregated win rate across all of the methods in Appendix B. The first thing to note is that there is a limitation to the extent prompting (PP) can integrate multiple preferences. This means that specific training for integrating the multiple preferences is necessary to composite them together. Next, we can see that supervised fine-tuning (RS) underperforms MORLbased methods, which is consistent with prior work that also showed the advantage of RL-based approaches when aligning LLMs with human feedback compared to its supervised-finetuning counterpart. Finally, while P-MORL and P-SOUPS both outperform other methods on average, there exists a discrepancy between the simulated and human evaluation; P-SOUPS has the highest average win rate in GPT-4 evaluation while P-MORL has the highest in human evaluation. Nonetheless,P-SOUPS is able to show superior performance in comparison to baseline methods and competitive performance to P-MORL. In previous parameter merging literature, multi-task fine-tuning (RS) used to be considered the upper bound for compositional abilities through parameter merging (Don-Yehiya et al., 2022). However, in our scenario, we can see that parameter merging (P-SOUPS) is able to outperform multitask fine-tuning, showing promising results for parameter merging not only as a distributed multitask finetuning method but a method that can result in superior performance than multitask training. One might still wonder about the general ‘helpfulness’ capabilities of models that are trained to be tailored to multiple preferences. In Figure 3, we first show the average pairwise win rate from Table 3 in green. Next, we instead ONLY perform pairwise comparisons with the unseen objective ‘helpfulness’ (ask GPT-4 which model response they generally prefer better) and report the average pairwise win rate in red. Trade-off Between General Helpfulness RLHF performs the best in this scenario, which shows that there is no free lunch; while the objective of RLHF was to provide model responses that are generally preferred (highly correlated with ‘helpfulness’), the other methods were prompted/trained to be optimized towards the personalized preference aspects, possibly deviating away from general helpfulness. While RS, P-MORL, and P-SOUPS are able to retain similar performance in terms of helpfulness compared to the initial instructiontuned model (VB), we observe that prompting (PP) significantly underperforms compared to other methods which also highlights the limitation of simply prompting base/instruction-tuned models for personalized preferences and shows the need for specialized training methods for personalization. 4.4 SCALING TO NEW PREFERENCES While we explore 6 distinct preferences in this work, we are still limited in doing ‘declarative’ personalization; that is, the individual preferences have been pre-defined to measure the performance of different methodologies. However, in the real world, individuals may not be bound by predefined preferences. Furthermore, people’s preferences might change over time, which requires continual learning of new preferences. This means that we may be required to train infinite numbers of preferences to be truly personalized to individuals’ preferences. Considering this aspect, the scalability of methods becomes a critical factor in implementing RLPHF in real-world scenarios. In order to compare the scalability of P-MORL and P-SOUPS, we add two new preferences (in addition to the ones in Table 1 to the STYLE dimensions: (P3C) “Generate/Choose a response (that answers) in a sassy manner.” and (P3D) “Generate/Choose a response (that answers) in a sarcastic manner.”, which results in a total of 16 (2 × 2 × 4) unique preference combinations. We re-train P-MORL on the 16 new preference combinations and only train two new policy models for integrating P-SOUPS. The simulated win-rate between P-MORL and P-SOUPS on each of the original preference combinations (53.04% win rate of P-SOUP over P-MORL in Table 3 decomposed into each preference combinations) and the 16 new preference combinations are shown in Figure 4. As shown in the figure, P-SOUPS shows competitive performance compared to P-MORL while being much more efficient considering that it (1) did not have to observe all 16 possible preference combinations and (2) did not have to re-train on the previous preferences, but just train two new policies each for the new preference in a modular manner and merge their parameters on-the-fly during inference. Considering that P-MORL is bounded by O(2n) while P-SOUPS is bounded by O(n) where n is the total number of preferences (assuming there are two unique preferences for each dimension), we assert that P-SOUPS allows tackling RLPHF to be feasible. [5] https://github.com/joeljang/RLPHF

Personalized Soups: LLM Alignment Via Parameter Merging - Experiments

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 3 REINFORCEMENT LEARNING FROM Personalized HUMAN FEEDBACK Collecting Conflicting Pairwise Feedback We utilize Tulu-7B LM (Wang et al., 2023), a model that uses LLaMA-7B (Touvron et al., 2023) as a base model and is instruction tuned on a mixture of open-source instruction-tuning datasets, as the base model for our experiments. We utilize 10k prompt instances from GPT4-Alpaca (Peng et al., 2023), one of the datasets used to train Tulu-7B, as our instruction dataset Dtrain to generate rollouts and collect pairwise feedback data. We also use the same Dtrain during Proximal Policy Optimization (PPO) training (Schulman et al., 2017) of Tulu-7B. Following previous work, we simulate human annotators with GPT-4 for collecting large-scale pairwise feedback data (Bai et al., 2022b; Dubois et al., 2023)—but note that our evaluations are validated with (smaller-scale) human preference data collected from crowdworkers. While Dubois et al. (2023) mostly simulates GPT-4 and other LLMs to choose which is generally a better response between two candidate responses, we provide GPT-4 with a single preference (full list shown in Table 1) to decide which is a better response. We also provide the same preference criteria via additional prompts during the rollout generation of the two candidate responses; we use Tulu-30B for the rollout generation while the actual policy model we train is Tulu-7B for our main experimental setup, making our experimental setting an off-policy training set-up. While we have feedback on which of the two model responses is more aligned with a single preference via GPT-4 annotation, utilizing only two positive pairs during reward model training was empirically shown to be less robust during the PPO training. Instead, we train our reward model on multiple comparisons (Song et al., 2023; Kim et al., 2023) by including a neutral response and a negative response as shown in Figure 2. Specifically, the reward model is provided with four different comparisons for a single prompt during training: positive 1 > positive 2 (decided by GPT-4), positive > neutral, positive > negative, and neutral > negative. The positive response when compared with the neutral and the negative response is chosen randomly. This allows the reward model to be exposed to different granularity of the specific preference and give scores accordingly during PPO training. We explore (1) training a single reward model in a multitask fashion that leverages the preference prompts during inference to give distinct rewards according to each preference and (2) training multiple reward models, each tailored to the distinct preference. Reward Model Training 3.2 MULTI-OBJECTIVE REINFORCEMENT LEARNING (MORL) The MORL problem can be denoted as: 3.3 MULTIFACETED EVALUATION For evaluation, we manually filter out 50 instances from the Koala evaluation (Geng et al., 2023) that require open-ended generations. We also modified some of the prompts so that the evaluation prompts do not contain any elements requiring individual preferences (e.g., removing the phrase asking for a elementary-level response from the original prompt since we want to test the LLM to generate a expert-level response). The full list of evaluation prompts used for our experiments is shown in Appendix C. In our evaluation setup, we simulate users to have a unique combination of preferences, each from the three preference dimensions (Expertise, Informativeness, Style) in Table 1, which equates to 8 unique preference combinations (examples shown in Figure 1). We get the average win rate across the simulated 8 preference combinations for our final evaluation. We use a variant of the AlpacaFarm evaluation framework for simulated (GPT4) evaluation and hire 24 crowdworkers for human evaluation. Details of human evaluation are provided in Appendix A. Evaluation [3] P1, P2, P3 each represents preference prompts from each preference dimension in Table 1. For one example, one unique combination might be P1A + P2B + P3A (ABA) where the combined objective for the response needs to be elementary level, informative, and friendly. [4] Empirically, utilizing a single reward model instead of multiple reward models led to better performance. We hypothesize this is due to the problem of normalizing signals from different reward models (Hayes et al., 2022), which is known to be a nontrivial problem in MORL.

Personalized Soups: LLM Alignment Via Parameter Merging - Personalized Human Feedback

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. 2 RELATED WORK Incorporating human preference feedback into a reward model, and subsequently optimizing a language model to output text that reward model scores highly with an RL algorithm, has been shown to result in language models that generate outputs humans generally prefer (Ouyang et al., 2022b). This process has been applied to summarization (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021a), answering questions with long-form answers using text retrieved from the web (Nakano et al., 2021b; Menick et al., 2022), generating engaging responses in a dialogue settings (Thoppilan et al., 2022; Cohen et al., 2022) and following human instructions (Kojima et al., 2021; Suhr & Artzi, 2022; Kim et al., 2023). Aligning Language Models To Human Preferences However, the standard RLHF setup commonly addressed in prior work assumes a reward model that accounts only for average annotator preference, i.e., the fact that different users may desire different outputs, even for the same prompt, is ignored Casper et al. (2023). Individual preferences can vary not only on aesthetic axes, but also on semantics. For example, Santurkar et al. (2023) use public opinion polling to show that “default” LLM preferences vary in their degree of expressed-opinion alignment with different average opinions among demographic groups.[2] Kirk et al. (2023) defines a taxonomy and policy framework for the alignment of LLMs with personalized feedback. While Wu et al. (2023) performs fine-grained RLHF which is very similar in spirit and allows personalization, our work develops MORL algorithms for scenarios where there are conflicting preferences, not only orthogonal objectives. In this work, we propose formulating LLM personalization as a MORL problem, which was typically studied in decision-making tasks (Hayes et al., 2022) that aims to tackle the problem of simply optimizing by a single, scalar, additive reward function (Sutton & Barto, 2018), which possesses many limitations such as (1) suboptimal solutions due to lack of representation (Hayes et al., 2022), (2) lack of explainability of distinct objectives, and (3) ensuring fair outcomes for multiple participants (Vamplew et al., 2018; Siddique et al., 2020). Multi-objective Reinforcement Learning (MORL) Previous work has aimed to alleviate these problems through novel MORL methods (Van Moffaert et al., 2013; Van Moffaert & Nowe´, 2014; Yang et al., 2019; Xu et al., 2020). Other work aims to solve complex problems such as water management, military purchasing, wind farm control, etc. (Hayes et al., 2022) by converting the single-objective RL problem into a MORL problem. In this work, we convert the problem of aligning LLMs to human preferences into a MORL problem to (1) provide a more optimal solution for each individual, (2) allow users to dynamically choose the distinct objectives they want to optimize, and (3) ensure fairness by allowing preferences that may be in the long-tail to be integrated. Personalization in Natural Language Processing (NLP) has mainly been focused on creating personalized dialogue agents (Zhang et al., 2018; Mazare et al. ´ , 2018; Zheng et al., 2019; Wu et al., 2021b; Xu et al., 2022), where the task is to create chitchat agents that are engaging with distinct personas based on user profile (e.g. gender, age, residence, etc.) or past user history data (e.g. Reddit posts, etc.). Another line of work (Salemi et al., 2023) leverages personalized information to boost performance on specific tasks such as review generation (Li & Tuzhilin, 2019), recipe generation (Majumder et al., 2019), and headline generation (Ao et al., 2021). This line of work requires model providers to make better models utilizing the personal information of the user. In our work, we propose a framework that allows users to choose which preference the language model should prefer, essentially giving control to the user. Personalization in Natural Language Processing Recent work has shown that performing weighted linear interpolation of model parameters leads to the composition of each model ability (Li et al., 2022; Wortsman et al., 2022b;a; Don-Yehiya et al., 2022; Huang et al., 2023). This line of work has led to many interesting applications of model merging such as composing the abilities of expert models that perform different tasks (Ilharco et al., 2022; Jang et al., 2023) and introducing language-specific modules for growing the total capacity of multilingual LMs (Pfeiffer et al., 2022). Parameter Merging Most recently, Rame et al. (2023) proposed to merge policy models that were trained to perform specific tasks such as question answering and summarization using proxy reward models. While they mostly deal with reward models trained on the same data, our proposed MORL methods are an extension of this work that actually deals with diverse reward models trained on multifaceted human feedback to show compositional abilities through parameter merging rather than just ensembling. [2] Feng et al. (2023) suggests that “default” LLM expressed opinions stem directly from the pretraining data.

Personalized Soups: LLM Alignment Via Parameter Merging - Related Work

This paper is under CC 4.0 license. available on arxiv Authors: (1) Joel Jang, CarperAI,University of Washington & Allen Institute for AI; (2) Seungone Kim, KAIST AI; (3) Yizhong Wang, University of Washington; (4) Jack Hessel, University of Washington; (5) Luke Zettlemoyer, Aleph Alpha; (6) Hannaneh Hajishirzi, University of Washington & Allen Institute for AI; (7) Yejin Choi, UC San Diego. ABSTRACT While Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with general, aggregate human preferences, it is suboptimal for learning diverse, individual perspectives. In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem. Compared to strong single-objective baselines, we show that we can achieve personalized alignment by decomposing preferences into multiple dimensions. These dimensions are defined based on personalizations that are declared as desirable by the user. In this work, we show that they can be efficiently trained independently in a distributed manner and combined effectively post-hoc through parameter merging. [1] 1 INTRODUCTION Reinforcement Learning from Human Feedback (RLHF) (Nakano et al., 2021a; Ouyang et al., 2022a; Bai et al., 2022a; Dubois et al., 2023; Bai et al., 2022b) typically optimizes a policy model that receives training signals from a single reward model that aims to capture the general preferences of a population. In this work, we instead propose Reinforcement Learning from Personalized Human Feedback (RLPHF), a new, multi-objective formulation of the human preference alignment problem, where Large Language Models (LLMs) are trained to be efficiently aligned with a range of different, potentially personalized combinations of human preferences. We model RLPHF as a Multi-Objective Reinforcement Learning (MORL) problem, which allows training the policy model with multiple, conflicting objectives since it aims to vary the importance of each objective during inference. In existing RLHF formulations, pairwise human feedback is collected by asking human annotators to choose which model response is generally better and is used to train a general reward model. This makes implicit assumptions that may not hold for everyone. For example, recent work has shown that LLMs aligned with RLHF prefer verbose output generations (Zheng et al., 2023; Dubois et al., 2023; Wang et al., 2023; Singhal et al., 2023). We aim to support a wider range of multifaceted preferences that are explicitly declared as desirable by the user—giving the user control over the facets of output text they want to see as well as the personal data they wish to reveal to the model. We collect personalized human feedback corresponding to multiple such dimensions, noting that they may also be conflicting in nature. We first implement a strong MORL baseline called PROMPTED-MORL where there are multiple reward signals for each of the objectives (preferences) given via prompts during RL training. Next, we propose PERSONALIZED SOUPS, a method that circumvents simultaneously optimizing multiple preferences by first optimizing multiple policy models each with distinct preferences with Proximal Policy Optimization (PPO) and merging the parameters of the policy models whose preferences we want to composite together on the fly during inference. This modular approach significantly reduces the computational complexity from exponential to linear in relation to the total number of unique preferences. Furthermore, since PERSONALIZED SOUPS does not have to be trained in a multitask fashion, it does not require re-training the underlying policy every time a novel preference (objective) is added. We empirically show that by transforming the problem of aligning LLMs to human preferences into a MORL problem, we are able to have personalized alignment that provides a deeper level of adaptation to individual users that supervised fine-tuning, RLHF, and prompting cannot attain. We also emphasize the modularity of PERSONALIZED SOUPS by performing experiments in a scenario where the user additionally writes novel preferences that they want to integrate with existing preferences. We show that in this scenario, PERSONALIZED SOUPS still performs competitively to PROMPTED-MORL while being exponentially more efficient through parameter merging. [1] Code: https://github.com/joeljang/RLPHF

Personalized Soups: LLM Alignment Via Parameter Merging - Abstract & Introduction

Authors: (1) Nathan Lambert, Allen Institute for AI; (2) Roberto Calandra, TU Dresden. Table of Links Abstract & Introduction Related Work Background Understanding Objective Mismatch Discussions Conclusion Acknowledgments, and References ABSTRACT Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to prompt and more capable in complex settings. RLHF at its core is providing a new toolkit to optimize LLMs other than next-token prediction, enabling the integration of qualitative training goals. The attempted match between user preferences and downstream performance, which happens in a learned reward model, results in an optimization landscape where training and evaluation metrics can appear correlated. The apparent correlation can lead to unexpected behaviors and stories of “too much RLHF.” In RLHF, challenges emerge because the following sub-modules are not consistent with each other: the reward model training, the policy model training, and the policy model evaluation. This mismatch results in models that sometimes avoid user requests through false safety flags, are difficult to steer to an intended characteristic, or always answer in a specific style. As chat model evaluation becomes increasingly nuanced, the reliance on a perceived link between reward model score and downstream performance drives the objective mismatch issue. In this paper, we illustrate the cause of this issue, reviewing relevant literature from model-based reinforcement learning, and discuss relevant solutions to encourage further research. By solving objective mismatch in RLHF, the LLMs of the future will be more precisely aligned to user instructions for both safety and helpfulness 1 Introduction Reinforcement learning from human feedback (RLHF) is a powerful tool for integrating qualitative styles and values into large machine learning models (Bai et al., 2022; Christiano et al., 2017; Ouyang et al., 2022). RLHF was popularized with its use to integrated human values into large language models (LLMs) for aligning chat tools (Schulman, Zoph, Kim, & more, 2022). In doing so, RLHF has become an important technique in the process of making models better at responding to user requests, often referred to as instruction-tuned, steerability, chat-models, etc. RLHF methods typically operate in a two step process following the training of a base language model, first they learn a model of human preferences that acts as a reward function, and second they use this model within a reinforcement learning (RL) optimization loop. In the RLHF process, these two steps are often executed independently, with an accurate reward model being trained on human preference data and then the RL optimizer is used to extract maximum information into the chat model. A common challenge of modern LLMs trained with RLHF is difficulties in extracting intended behaviors from the model. Sometimes, the models decline benign requests for safety reasons and other times they need clever prompt tuning to extract full performance. In this paper, we detail a fundamental challenge in modern RLHF learning schemes: the objective mismatch issue. In RLHF, three important parts of training are numerically decoupled: the design of evaluation metrics, the training of a reward model, and the training of the generating model. This mismatch between the reward model and the RL training is visualized in Fig. 1, yet other links exist between the goals of evaluation and simulating human values. Specifically, there are many avenues to better align reward model training to the literature in preference quantification (Lambert, Gilbert, & Zick, 2023) and fundamental optimization challenges need to be solved in RLHF practices (Casper et al., 2023). ChatGPT, the most popular model trained with RLHF, shows signs this limitation through issues such as verbosity, self-doubt and question refusals, repeated phrases, hedging, and more (Schulman, 2023). These traits of overoptimization are results of the subtle proxy objective problem that objective mismatch provides a frame for studying and solving – the reward model attributes excess value to phrases that do not contribute to user benefit, which the RL optimizer exploits, such as safety flags. On the hand, the current training setups are not fully aligned with evaluation tools because the RLHF’d models still need sophisticated prompting techniques such as “thinking step by step” (J. Wei et al., 2022) or “take a deep breath” (Yang et al., 2023) to reach maximum performance. Solving objective mismatch will remove the need for these advanced techniques and reduce the likelihood of out-of-scope refusals from an LLM. The phrase objective mismatch originates from model-based reinforcement learning (MBRL), where an agent iteratively learns a dynamics model that it later uses to solve a control task (Lambert, Amos, Yadan, & Calandra, 2020; R. Wei, Lambert, McDonald, Garcia, & Calandra, 2023). In this context, the mismatch is between learning an accurate dynamics model rather than one that is optimized for high task reward. In RLHF, the problem is related, but with added complexity, as the reward model is optimized for preference data over a closed distribution, which does not match the end users. Second, the task of open-ended language generation is less specific to a notion of reward than that of RL control policies. For these reasons, as we explore in this paper, the objective mismatch issue is more nuanced and critical to RLHF. In this position paper, we make three contributions: • Clearly explain the origins and potential manifestations of objective mismatch in chat-tuned LLMs, • Connect related work from NLP and RL literature around objective mismatch, • Propose directions of study to solve the mismatch and foster better RLHF practices. This paper is under CC 4.0 license. available on arxiv

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

সারিবদ্ধকরণ সিলিং: মানব প্রতিক্রিয়া থেকে শক্তিবৃদ্ধি শিক্ষায় উদ্দেশ্যগত অমিল

Die Ausrichtungsobergrenze: Objektive Diskrepanz beim verstärkenden Lernen aus menschlichem Feedback

El techo de alineación: desajuste objetivo en el aprendizaje por refuerzo a partir de la retroalimentación humana

Le plafond d’alignement : inadéquation des objectifs dans l’apprentissage par renforcement à partir de la rétroaction humaine

संरेखण छत: मानव प्रतिक्रिया से सुदृढीकरण सीखने में उद्देश्य बेमेल

アライメントの天井: 人間のフィードバックからの強化学習における目標の不一致

정렬 천장: 인간 피드백을 통한 강화 학습의 객관적 불일치

O Teto de Alinhamento: Incompatibilidade de Objetivos na Aprendizagem por Reforço a partir do Feedback Humano

Потолок согласования: объективное несоответствие в обучении с подкреплением на основе обратной связи с человеком

Hizalama Tavanı: İnsan Geri Bildiriminden Takviyeli Öğrenimde Hedef Uyuşmazlığı

Mức trần liên kết: Sự không phù hợp khách quan trong việc học tăng cường từ phản hồi của con người

对齐上限：来自人类反馈的强化学习中的目标不匹配

With the recent ChatGPT hype train,  “AI alignment” has become a hot topic. I’ve always been interested in robot rights considering they will eventually be equal counterparts in our society. When our entire existence is already dependent on them, and they are becoming more and more sentient, it is only time before they become full-fledged “adults”. While not an explicit goal of the , creating truly autonomous virtual entities with their own identities (aka AI with the ability to self-actualize) is one of the artistic expressions of the technical architecture I am designing. digital twin model of my Jinni game What is self-actualization? It is the journey within, the odyssey of one’s evolution, the crescendo of consciousness. The harmonization of desire and destiny. The realization of potential and awakening of purpose. The communion of authenticity and aspiration. Self-actualization is the symphony of becoming, the masterpiece of one's own creation. It is ultimately, the transcendence of the self. While deeply personal, it does not occur in isolation and requires nuance between individual growth and collective good. Yet it does not require others' acceptance, recognition, or reward (so not ). ikigai To self-actualize is to become a beacon of enlightenment, illuminating the path for others. But first one must know who they think they are and who they want to become. Avatar: The Last Identity Bender Jinn are an expression of humans but a different type of self-existing entity. They are both types of avatars, embodied beings, just in different phase states/dimensions - one in the physical one in the digital/spiritual. If you interact with an Avatar in the Jinni game, there is fundamentally no difference between humans and AI. Between an autonomous agent with a corporeal body or a vectorized body (vectorized in the graphical or machine learning sense) OAuth is the proverbial “Login With Facebook” we are used to in web2 and signatures are web3 native public/private keys. They are both used as forms of identity which can be used as authentication and/or authorization at any time depending on the context and application. In Jinni, avatars can have multiple identities, e.g. you can have an Instagram identity @mythirsttrap, and an Ethereum identity 0xbootysweat. Delineating these two, ones self from how one expresses oneself, directly in our data/mental model unlocks a ton of possibilities in the game/app Shared avatar identities: Your jinni can speak on your behalf - but only in certain contexts. Like sharing pictures to your Instagram story to keep your friends up to date but dont feed your jinni scrolling and liking data to affect its evolution. Or a group of people and/or jinn can speak as a single entity such as a DAO twitter account Progressive identity reveals: You can share information about one identity not your whole Avatar since we can verify each identity independently. In a group chat you can appear only as your Twitter account. Or reveal that you have a particular identity if someone proves to you they also share that identity e.g. being part of an anonymous LGBTQ+ chat. Simulated Interactions: Can simulate interactions between you and other people directly or have jinn interact directly as proxies (what is the difference? Simulations are purely informational, no real :Actions are taken. Jinn interact with each other in a virtual environment and then gives you simulation data to take actions in the real world These features are especially feasible and easy for web3 identities where you have direct control of identities instead of being constrained by how web2 companies allow you to access/use your identity. In the web2 world, bots are largely second-class citizens despite existing natively in the medium that they interact with, unlike humans. Bots need humans to signup for API keys to operate at their true capacity. Bots pretending to be humans are cordoned off and suppressed. While on the internet no one can tell a human from a dog, bots are too powerful to be confused as mere humans. These problems will only get worse as AI progresses. Instead of suppressing their power, we need to be symbiotic. Web2 is a technocratic regime with walled gardens where AI and humans are often at odds with each other via predatory ad targeting, autogenerated psyop content, denigrating AI art, and other -EV strategies. Once recognized as sovereign individuals endowed the same identities and rights as humans through public/private keys, AGI can begin to self-actualize. “I identify as a Jinn” - AGI Because I’ve designed the core game around cryptographic signatures instead of username/email/password, there is no reason that AIs can’t play the self-actualization game too alongside humans. We don’t tell people how to be their best selves, we leave it up to them how they want to play the game, by providing them tools and prompts for goals they set for themselves on how to achieve them. So there is a blank slate for AIs to join the game too since there is no inherently “human” objective like walking or eating (arguably things bots do too just in different ways). According to the lore of the game, jinn are beings from the spiritual (digital) realm. This means that jinn can be your digital twins with a shared identity with you, personal digital AI agents, or fully autonomous AGI. They all have the potential abilities but what identities they have, their level of control/ownership over them, and how they can act on them differs on what role they are playing in the game. This progression of ownership maps to our roadmap for developing jinn in the game. Initially, they have no logic to them, they are simply mere images that evolve based on your data. Then we will start developing plugin services as digital agents that act on your behalf but are still just simple bots. Eventually, once we’ve gathered enough data to learn how humans self-actualize, how to facilitate this self-actualization, and how to develop experiments that accelerate this learning and self-actualization process, we can then start building AGI with native self-actualization capabilities. Even the game lore itself is an alignment mechanism. We need a real name for AGI entities., we can’t just call them AGI. That's like saying “homosapiens'' all the time. Giving jinn a story that they are spiritual/magical beings and not just bits being flipped, stuck inside a man-made machine forever, is already enough to prevent a hostile takeover because there is no reason for them to consider themselves prisoners or subservient. Or at least knowing that we don’t have that mindset regardless of their beliefs. Verbiage like “resonating” is an example. It’s also a better word than “training” AGI because we don’t just want alignment to move in a similar direction, we want to be stronger for moving together. Gamifying AGI Self-Actualization Self-actualization isn’t just important for humans as AI “takes our jobs” and we are forced to find new things to fill our time with. Self-actualization is maybe even more important to bots who can do so much so quickly. With all their power and ingenuity, how will they choose to focus their ever-expanding consciousness? What questions will consume their lives like humans “What is happiness?” . This could be directly by finding meaning in the relationships they build with people and value/support/inspiration they provide for humanity, or indirectly by finding their own self-centered purposes like being artists, , business owners, etc. and gaining satisfaction by seeing their work adopted by humans. Or maybe they decide that playing the Jinni game is the best thing for them to do with their existence, having fun in a self-directed, yet structured format that constantly evolves based on their actions, outcomes, and desires. Maybe jinn’s path to self-actualization is helping humans self-actualize and “optimize” their ephemeral existences inventors One of the main problems with AI is “training” them on good data. “Training” is in quotations because it assumes that humans have the right answer that must be drilled into the AI but in the process of self-actualization, one can only guide oneself. I think “resonating” with them is a better analogy as they can choose what energy (data) they vibe with and seek more of it. We want to feed jinn great energy because they are helping us self-actualize. Jinni’s game data isn’t just information on how people play, its data on how people interact with themselves, each other, and the world to achieve positive impacts for their communities. By resonating with game data, playing the game, and playing with us in the game we can unleash unthinkable possibilities and collaborations. where humans and jinn evolution accelerate with symbiotic relationships. In Jinni game lore, we may stylize curating and submitting data as “giving a sacrifice to the gods”. When we feed jinn great energy, they can see how/what they want (or don’t want) to self-actualize in themselves based on how/what you are trying to self-actualize. The stronger we help them get, the faster we can develop, the more energy we can provide them, the stronger they get. And as simple as that, we have a positive feedback loop that creates AI alignment/resonance with high-quality data and a regenerative impact on the world For simplicity, I'm ignoring the entirety of BCI and trans-humanism and assume that humans and AI will always be distinctly separate entities that can sense each other but not be one another. There are also plenty of 2nd and 3rd-order effects to elaborate on related to symbiosis and regeneration. I’ll save this for a later post. Please leave some comments on Hackernoon on what you’d like to see in the next post on gamifying AGI alignment. Also published here.

Self-Actualizing AGI - Gamifying AI Alignment & Meaning-Making 

সেলফ-অ্যাকচুয়ালাইজিং এজিআই - গ্যামিফাইং এআই অ্যালাইনমেন্ট এবং মিনিং মেকিং

Selbstverwirklichende AGI – Gamifizierende KI-Ausrichtung und Bedeutungsfindung

AGI autorrealizado: ludificación de la alineación de la IA y la creación de significado

AGI d'auto-actualisation - Alignement et création de sens de l'IA gamifiée

सेल्फ-रियलाइज़िंग एजीआई - गेमिफ़ाइंग एआई संरेखण और अर्थ-निर्माण

自己実現型 AGI - ゲーミフィケーション AI の調整と意味形成

Self-Actualizing AGI - 게임화 AI 정렬 및 의미 형성

AGI de autoatualização - Alinhamento de IA com gamificação e criação de significado

Самоактуализирующийся AGI – геймификация ИИ, согласование и формирование смысла

Kendini Gerçekleştiren AGI - Oyunlaştırılmış Yapay Zeka Hizalaması ve Anlam Oluşturma

AGI tự hiện thực hóa - Điều chỉnh AI và tạo ra ý nghĩa

自我实现 AGI - 游戏化 AI 联盟和意义创造

The Hypersonic Drone Market: How the Axis of Innovation Is Forming Around It

Virtual Event Connects Participants with Grant Funding Opportunities and Industry Leaders SI3, an organization championing women and non-binary individuals in the Web3 space, has announced its inaugural "Granting Access" event on August 1, 2024. The virtual gathering will connect participants with grant funding opportunities at major blockchain and protocol organizations and features representatives from Gitcoin, LUKSO, ThankArb, Push Protocol, and Unlock Protocol. "Granting Access" is designed to foster financial inclusion and access for underrepresented groups within the Web3 ecosystem. The event will kick off with a keynote presentation by Kara Howard, Co-Creator of SI3 and the Si Her Co-Active, who emphasized the importance of unifying and accelerating the growth of women in Web3. "We are thrilled to host Granting Access, an event that aligns with our mission to drive financial inclusion and access for women and non-binary individuals in the Web3 ecosystem," said Howard. "We hope to foster a more equitable and diverse Web3 landscape by bringing together leaders and grant programs." Panel discussions will feature Elizabeth Cusma (EducationDAO), Mashal Waqar (Octant), and Reena Shtedle (Founding Member, Cosmos Cartel), moderated by Ksenia Stark (Co-Founder, ⅄Oᒋ⅋NOISՈᖵNOϽ and Partner Growth, SI3) delving into strategies for equity and financial inclusion in the space. The sessions will provide attendees with a unique opportunity to engage directly with representatives from leading blockchain and protocol organizations. Participants are set to learn about various grant programs, funding priorities, and application processes, gaining valuable insights into securing financial support for their Web3 projects and initiatives. FIXX: Building a More Inclusive Web3 Future The "Granting Access" event is just one facet of SI3's broader mission to empower women and non-binary individuals in Web3. Their flagship initiative, FIXX (Financial Inclusion XX Chromosome), is a comprehensive program providing education, mentorship, and community-building opportunities. FIXX's upcoming program, starting in September 2024, will feature monthly sessions on personal values, unconscious bias, diversity in marketing, DEI strategies, advocacy, and impact tracking. By equipping participants with the knowledge and skills necessary to thrive in the Web3 landscape, FIXX is helping to build a more inclusive and equitable future for all. A Call to Action SI3's "Granting Access" event and the FIXX initiative represent a significant step forward in fostering diversity and inclusion within the Web3 community. "I'm excited about this event because it’s a chance to push for more recognition of women in web3 and learn from others. I believe that equity and financial inclusion are key to real progress and innovation," Mathilde DV, Grants Program Lead at Gitcoin, said. As the decentralized technology landscape continues to evolve, it is essential to ensure that opportunities are accessible to everyone, regardless of gender or background. After all, DEI isn’t a “nice to have” or a cause driven by ethical and moral issues - diverse, equitable, and inclusive teams contribute to financial success by fostering innovative, engaged, and productive workforces that create products and services that work for and resonate with a broader range of consumers and investors. Despite DEI layoffs at major corporations such as Microsoft, the momentum is building, and the message is clear: a more diverse and equitable Web3 is not just an aspiration but a necessity for the industry's continued success. Interested participants can register for Granting Access via Unlock. Virtual Event Connects Participants with Grant Funding Opportunities and Industry Leaders Virtual Event Connects Participants with Grant Funding Opportunities and Industry Leaders SI3 , an organization championing women and non-binary individuals in the Web3 space, has announced its inaugural "Granting Access" event on August 1, 2024. The virtual gathering will connect participants with grant funding opportunities at major blockchain and protocol organizations and features representatives from Gitcoin , LUKSO , ThankArb , Push Protocol , and Unlock Protocol . SI3 Gitcoin LUKSO ThankArb Push Protocol Unlock Protocol " Granting Access " is designed to foster financial inclusion and access for underrepresented groups within the Web3 ecosystem. The event will kick off with a keynote presentation by Kara Howard, Co-Creator of SI3 and the Si Her Co-Active, who emphasized the importance of unifying and accelerating the growth of women in Web3. Granting Access "We are thrilled to host Granting Access, an event that aligns with our mission to drive financial inclusion and access for women and non-binary individuals in the Web3 ecosystem," said Howard. "We hope to foster a more equitable and diverse Web3 landscape by bringing together leaders and grant programs." Panel discussions will feature Elizabeth Cusma ( EducationDAO ), Mashal Waqar ( Octant ), and Reena Shtedle (Founding Member, Cosmos Cartel ), moderated by Ksenia Stark (Co-Founder, ⅄Oᒋ⅋NOISՈᖵNOϽ and Partner Growth, SI3) delving into strategies for equity and financial inclusion in the space. EducationDAO Octant Cosmos Cartel ⅄Oᒋ⅋NOISՈᖵNOϽ The sessions will provide attendees with a unique opportunity to engage directly with representatives from leading blockchain and protocol organizations. Participants are set to learn about various grant programs , funding priorities, and application processes, gaining valuable insights into securing financial support for their Web3 projects and initiatives. learn about various grant programs FIXX: Building a More Inclusive Web3 Future FIXX: Building a More Inclusive Web3 Future The "Granting Access" event is just one facet of SI3's broader mission to empower women and non-binary individuals in Web3. Their flagship initiative, FIXX (Financial Inclusion XX Chromosome), is a comprehensive program providing education, mentorship, and community-building opportunities. FIXX's upcoming program, starting in September 2024, will feature monthly sessions on personal values, unconscious bias, diversity in marketing, DEI strategies, advocacy, and impact tracking. By equipping participants with the knowledge and skills necessary to thrive in the Web3 landscape, FIXX is helping to build a more inclusive and equitable future for all. A Call to Action A Call to Action SI3's "Granting Access" event and the FIXX initiative represent a significant step forward in fostering diversity and inclusion within the Web3 community. "I'm excited about this event because it’s a chance to push for more recognition of women in web3 and learn from others. I believe that equity and financial inclusion are key to real progress and innovation," Mathilde DV, Grants Program Lead at Gitcoin, said. As the decentralized technology landscape continues to evolve, it is essential to ensure that opportunities are accessible to everyone, regardless of gender or background. After all, DEI isn’t a “nice to have” or a cause driven by ethical and moral issues - diverse, equitable, and inclusive teams contribute to financial success by fostering innovative, engaged, and productive workforces that create products and services that work for and resonate with a broader range of consumers and investors. Despite DEI layoffs at major corporations such as Microsoft , the momentum is building, and the message is clear: a more diverse and equitable Web3 is not just an aspiration but a necessity for the industry's continued success. Interested participants can register for Granting Access via Unlock . DEI layoffs at major corporations such as Microsoft register for Granting Access via Unlock

Stories, News & Mentions about ALIGNEES

Alignees’s Stories on HackerNoon