Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 8 DISCUSSION 8.1 Supporting Users in Clarifying and Iterating Over Principles Finding the right granularity for a principle was sometimes challenging for participants; they created under-specified principles that did not impact the conversation, as well as over-specified principles that applied too strictly to certain portions of the conversation. One way to support users in finding the right granularity could be generating questions to help them reflect on their principle. For example, if an abstract principle is written (e.g., “Ask follow up questions to understand the users preferences”), an LLM prompt can be used to pose clarifying questions, such as, “What kind of follow up questions should be asked?” or, “Should I do anything differently, depending on the user’s answers?” Users could then answer these questions the best they can and then the principle could be updated automatically. Alternatively, another way to help users reflect on their principles might be to engage in a side conversation with a chatbot to help clarify them. This chatbot could pose similar questions as those suggested above, but it might also provide examples of chat snippets that adhere to or violate the principle. As they converse, the chatbot might continue to pose clarifying questions, while the principle is updated on the fly. Thus, future work could examine supporting users in interactively reflecting upon and clarifying their principles. 8.2 Organizing Principles and Supporting Multiple Principle Writers As participants accumulated principles, it was increasingly likely that there was a conflict between two of them, and it was harder for them to get an overview of how their principles were affecting the conversation. One way to prevent potential principle conflicts is to leverage an LLM to conduct a pairwise comparison of principles to assess if any two are at odds, and then suggest a solution. This kind of conflict resolution, while useful for a single principle writer, would be crucial in cases when multiple individuals are writing principles to improve a model. Multiple principle writers would be useful for gathering feedback to improve the overall performance of the model, but with many generated principles, it is increasingly important to understand how they might impact the conversation. Perhaps a better way to prevent conflicts is to organize principles in a way that summarizes their impact on the conversation. Principles are small bits of computation; there are conditions when they are applicable, and depending on those conditions, the bot’s behavior might branch in separate directions. One way to organize principles is to construct a state diagram, which illustrates the potential set of supported user flows and bot responses. With this overview, users could be made better aware of the overall impact of their principles, and they could then easily revise it to prevent conflicts. Therefore, another rich vein of future work is developing mechanisms to resolve conflicts in larger sets of principles, as well as organizing them into an easily digestible overview. 8.3 Automatically Ideating Multiple User Journeys to Support Principle Writing To test the chatbot and identify instances where the model could be improved, participants employed a strategy where they went through different user journeys with the chatbot. Often their choice of user journeys were biased toward what they were interested in, or they only tested the most common journeys. One way to enable more robust testing of chatbots could be to generate potential user personas and journeys to inspire principle writers. Going further, these generated user personas could then be used simulate conversations with the chatbot being tested [24]. For example, for VacationBot, one user persona might be a parent looking for nearby, family friendly vacations. A dialogue prompt could be generated for this persona and then VacationBot and this test persona could converse for a predefined set of conversational turns. Afterwards, users could inspect the conversation, and edit or critique VacationBot’s responses to generate principles. This kind of workflow could sidestep the challenge of repeatedly shifting from an end-user’s perspective to a bot-designer’s perspective, which exists in the current workflow. At the same time, users would be able to evaluate fuller conversational arcs, as opposed to single conversational turns. Thus, another line of future work is supporting users in exploring diverse user journeys with their chatbot, as well as exploring workflows that require less perspective switching. 8.4 Limitations and Future Work A set of well-written principles is often not enough to robustly steer an LLM’s outputs. As more principles are written, an LLM might “forget” to apply older principles [45]. This work focuses on helping participants convert their intuitive feedback into clear principles, and we illustrate that the principle-elicitation features help with that process. However, in line with the original Constitutional AI workflow [1], future work can focus on using these principles to generate a fine-tuning dataset, so that the model robustly follows them. Next, while we selected chatbots for two very common use cases for the study (vacation and food), participants might not have been very knowledgeable or opinionated in these areas. Future work can explore how these principle-elicitation features help users when writing principles for chatbot use cases that they are experts in. That being said, it was necessary to choose two chatbot use cases for the study to enable a fair comparison across the two conditions. This paper is under CC 4.0 license. available on arxiv

Interactively Critiquing LLMs by Converting Feedback into Principle— Conclusion and References

The Future of Chatbot Customization

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Detailed Analysis of Inter-Annotator Agreement

The Noonification: The FBI, Apple, and the San Bernardino Massacre (10/3/2023)

A Detailed Analysis of Inter-Annotator Agreement

A Detailed Analysis on the Effectiveness of Automatic Filtering

AI-Driven Creativity: QDAIF Shines in Generating Diverse and High-Quality Texts

A Detailed Analysis of Inter-Annotator Agreement

The Noonification: The FBI, Apple, and the San Bernardino Massacre (10/3/2023)

A Detailed Analysis of Inter-Annotator Agreement

A Detailed Analysis on the Effectiveness of Automatic Filtering

AI-Driven Creativity: QDAIF Shines in Generating Diverse and High-Quality Texts

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps