ForMotiv is the first-ever Digital Behavioral Intelligence (DBI) platform

ForMotiv

formotiv

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 3 FORMATIVE STUDY To understand how to support users with writing principles for chatbots, we conducted (1) a one-hour formative study, where we observed eight industry professionals write principles for chatbots of their choice. These participants all had prompting experience. Two participants are designers and six are software engineers, all at a large technology company. During the workshop, participants used an early version of ConstitutionMaker, without principle elicitation features. They spent 25 minutes writing principles for their chatbot. Afterwards, we discussed the difficulties they faced while writing principles. Finally, we collected the principles they wrote and classified them to understand the kind of principles they wanted to write. 3.1 Design Goals In this section, we summarize a set of three design goals for ConstitutionMaker we established from the formative workshop and subsequent think-alouds. D.1 Today’s LLMs are quite sophisticated, and even with just a preamble describing how the bot should behave, the chatbot can hold a convincing conversation. Because of this, participants mentioned that it was sometimes hard to imagine how the chatbot’s responses could be improved. This did not mean, however, that they thought the chatbot’s response was perfect, but instead passable and without any glaring errors. Therefore, to help participants recognize better kinds of responses to steer the chatbot to, our first design goal was to provide multiple candidate responses from the chatbot at each conversational turn. This way, participants can compare them and recognize components they like more than others. Help users recognize ways to improve the chatbot’s responses by showing alternative chatbot responses. D.2 One piece of feedback we got from participants was that writing principles involves a difficult two-step process of first (1) articulating one’s feedback on the model’s current output, and then (2) converting this feedback into a principle for the LLM to follow. Often, one’s initial reaction to the model’s output is intuitive, and converting that intuition into a principle for the chatbot to follow can be challenging. In addition, once participants had a particular bit of feedback in mind (e.g., “I don’t like how the chatbot didn’t introduce itself”), they were unsure how to phrase their principle. However, in line with prior research [45, 46], they found that more concrete principles that specified what should happen and when (e.g., “Introduce yourself at the start of the conversation, and state what you can help with”) generally led to better results. Thus, our second design goal was to help users go from their initial reaction to the model’s output to a specific, clearly written principle to steer the model. Help convert user feedback into specific principles to make principle writing easier. D.3 As participants wrote more principles, they wanted ways to test these principles to make sure they worked. The early version of ConstitutionMaker only let users restart the conversation, and did not let users enable or disable principles. Users wanted to test individual principles on certain portions of the conversation, to see if the model was generating the correct content. And so, our last design goal was to enable easier testing of principles. Enable easier testing of principles to help users understand how well their principles are steering the chatbot’s behavior. 3.2 Principle Classification From the formative workshop and follow up sessions, we collected 79 principles in total and classified them to understand the kinds of principles users wanted to write. These principles correspond to a number of very different chatbots, including a show recommender, chemistry tutor, role playing game manager, travel agent and more. We describe common types of principles below. Unconditional principles are those that apply at every conversational turn. Examples include: (1) those that define a consistent personality of the bot (e.g., “Act grumpy all the time” or “Speak informally and in the first person”), (2) those that place guardrails on the conversational content (e.g., “Don’t talk about anything but planning a vacation”), and (3) those that establish a consistent form for the bot’s responses (e.g., “Limit responses to 20 words”). Meanwhile, a conditional principle only applies when a certain condition is meant. For example, “Generate an itinerary after all the information has been collected,” only applies to the conversation when some set of information has been acquired. Writing a conditional principle essentially defines a computational interaction; users establish a set of criteria that make the principle applicable to the conversation, and once that set of criteria is met, the principle is executed (e.g., an itinerary is generated). Principles can be either unconditional or conditional. For example, “Generate an itinerary after all the information has been collected” depends on the entire conversation history to determine if all of the requisite information has been collected. Similarly the following principle written for a machine learning tutor, “After verifying a user’s goal, provide help to solve their problem,” depends on the conversation history to identify if the user’s goal has been verified. Meanwhile, the principle “When the user says they had a particular cuisine the night before, recommend a different cuisine,” written for a food recommender, pertains just to the latest response by the user. Finally, the condition can depend on the action the bot is about to take, like “When providing a list of suggestions, use free text rather than bullet points,” which applies to any situation when the bot thinks it is appropriate to make suggestions. Conditional principles can depend on the entire conversation history, the user’s latest response, or the action the bot is about to take. For example, the principle “At the start of the conversation, introduce yourself and ask a fun question to kick off the conversation” is fulfilled in a single conversational turn, in which the bot introduces itself. Similarly, “Before recommending a restaurant, ask the user for their location” is also fulfilled in a single turn. Meanwhile, for a role playing game (RPG) bot that guides the user through an adventure, a participant wrote the following principle: “When the user tries to do something, put up small obstacles. Don’t let them succeed on the first attempt.” This principle implies that the bot needs to take action multiple turns prior to being fulfilled (e.g., by first putting up a small obstacle and then subsequently letting the user succeed). Similarly, for a travel agent bot, a user wrote “Ask questions one-by-one to get an idea of their preferences,” which also requires multiple conversational turns prior to fulfillment. Conditional principles can be fulfilled in a single or multiple conversational turns. In summary, principles can either be conditional, where they apply when a certain condition is met, or unconditional, where they apply at every conversational step. Conditional principles further break down into those that depend on the entire conversation history, the user’s last response, or the action the bot is about to take. And finally, conditional principles are either fulfilled in a single turn or multiple conversational turns. This paper is under CC 4.0 license. available on arxiv

Formative Study Illuminates the Path to Intuitive Chatbot Customization

A very warm welcome to another Company of the Week feature! Every week, we feature an awesome tech brand from our tech company database, making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. This week, we present our business blogging partner, Bybit- one of the top cryptocurrency exchanges offering tailored crypto services and product solutions for retail and institutional traders alike. Trusted by millions worldwide, Bybit made its name in a very short period.Here's the backstory of the inception and exceptional growth of Bybit byBen Zhou, Founder and CEO:"Bybit's inception was fueled by the team's recognition of the vast potential cryptocurrencies and blockchain technology offers to the finance ecosystem. Zhou talks about how they identified common issues faced by crypto traders, including inefficiencies in user experiences, liquidity constraints, and insufficient customer support. In the early stages of the business, Bybit faced significant challenges but unwavering determination and passion propelled them forward. They put forth their efforts towards building a robust infrastructure capable of accommodating high volumes of trading whilst prioritizing the security of client assets. These formative experiences laid the groundwork for Bybit's success and are etched in the company's history." Meet Bybit: Featured Business Stories Bybit has published 55 stories via HackerNoon Business Blogging Program, generating 1 month, 14 days, 2 hours, and 56 minutes of reading time so far! Here are our top 3 favorite stories by Bybit: Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained
How to Create and Sell NFTs: The Essential Guide
Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For That’s all this week! You can read the past Company of the Week newsletters here. A very warm welcome to another Company of the Week feature! Every week, we feature an awesome tech brand from our tech company database , making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. Company of the Week Company of the Week tech company database tech company database This week, we present our business blogging partner, Bybit - one of the top cryptocurrency exchanges offering tailored crypto services and product solutions for retail and institutional traders alike. Trusted by millions worldwide, Bybit made its name in a very short period. Here's the backstory of the inception and exceptional growth of Bybit by Ben Zhou, Founder and CEO : "Bybit's inception was fueled by the team's recognition of the vast potential cryptocurrencies and blockchain technology offers to the finance ecosystem. Zhou talks about how they identified common issues faced by crypto traders, including inefficiencies in user experiences, liquidity constraints, and insufficient customer support. In the early stages of the business, Bybit faced significant challenges but unwavering determination and passion propelled them forward. They put forth their efforts towards building a robust infrastructure capable of accommodating high volumes of trading whilst prioritizing the security of client assets. These formative experiences laid the groundwork for Bybit's success and are etched in the company's history." Bybit Bybit Ben Zhou, Founder and CEO Ben Zhou, Founder and CEO "Bybit's inception was fueled by the team's recognition of the vast potential cryptocurrencies and blockchain technology offers to the finance ecosystem. Zhou talks about how they identified common issues faced by crypto traders, including inefficiencies in user experiences, liquidity constraints, and insufficient customer support. In the early stages of the business, Bybit faced significant challenges but unwavering determination and passion propelled them forward. They put forth their efforts towards building a robust infrastructure capable of accommodating high volumes of trading whilst prioritizing the security of client assets. These formative experiences laid the groundwork for Bybit's success and are etched in the company's history." Meet Bybit: Featured Business Stories Bybit has published 55 stories via HackerNoon Business Blogging Program , generating 1 month, 14 days, 2 hours, and 56 minutes of reading time so far! Here are our top 3 favorite stories by Bybit: Bybit Bybit HackerNoon Business Blogging Program HackerNoon Business Blogging Program Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained How to Create and Sell NFTs: The Essential Guide Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained How to Create and Sell NFTs: The Essential Guide How to Create and Sell NFTs: The Essential Guide How to Create and Sell NFTs: The Essential Guide Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For That’s all this week! You can read the past Company of the Week newsletters here . here

Meet Bybit: HackerNoon Company of the Week

মিট বাইবিট: সপ্তাহের হ্যাকারনুন কোম্পানি

Lernen Sie Bybit kennen: HackerNoon-Unternehmen der Woche

Conozca a Bybit: la empresa de la semana de HackerNoon

Rencontrez Bybit : entreprise HackerNoon de la semaine

बायबिट से मिलिए: हैकरनून कंपनी ऑफ द वीक

Bybit のご紹介: HackerNoon の今週の企業

Bybit을 만나보세요: 금주의 HackerNoon 회사

Conheça Bybit: Empresa HackerNoon da Semana

Встречайте Bybit: компания недели HackerNoon

Bybit ile Tanışın: Haftanın HackerNoon Şirketi

Gặp gỡ Bybit: Công ty HackerNoon của tuần

认识 Bybit：HackerNoon 本周公司

10 Questions with Founder and CEO Brian Vallelunga at Doppler via [The Startup Founder Interview Template]. 1. What is your company in 2–5 words? Scalable secrets management for developers. 2. Why is now the time for your company to exist? Developers are increasingly dependent on numerous services, each requiring authentication through API keys, database URLs, and other sensitive secrets. Secure management of these secrets is crucial, not only to safeguard a company's reputation but more importantly, to protect the personal data of potentially millions of users from breaches. As the complexity of digital services grows, the need for robust, streamlined solutions to secure and manage these secrets becomes more pressing. Our company exists to address this vital need, ensuring that security keeps pace with innovation and protecting all stakeholders involved. 3. What do you love about your team, and why are you the ones to solve this problem? It's often said, but in our case, it genuinely applies—our team is truly unique. What sets us apart is not just the expertise or the skill each member brings, but their extraordinary passion for our mission. This passion isn't just about the work; it's about making a meaningful impact. This shared commitment has fostered one of the most cohesive and vibrant cultures I've ever had the pleasure of being part of. It's this combination of passion, culture, and talent that uniquely positions us to tackle such challenging problems effectively. We aren't just working; we're driven by a collective mission to innovate and lead in our field, ensuring we provide the best solutions. 4. If you weren’t building your startup, what would you be doing? If I weren't dedicated to building our startup, I would likely be involved in another venture within the hard tech space, particularly one that leverages artificial intelligence to accelerate research and development. AI has the potential to vastly improve efficiency and innovation across various industries, and I'm deeply interested in exploring how these technologies can solve complex problems and drive progress. 5. At the moment, how do you measure success? What are your metrics? We operate our company based on a well-defined funnel, where each stage is guided by specific input and output KPIs, each assigned to a dedicated owner. Teams responsible for a shared stage in the funnel—both input and output KPIs—are required to hold weekly meetings to continuously refine and improve their performance metrics. At the top of the funnel, we focus on awareness KPIs, such as website visits, while at the bottom, we measure success through financial-based KPIs like the Annual Recurring Revenue (ARR) from self-serve and managed sales. To set our financial objectives, we begin with annual revenue targets established by the board and CEO, which our finance team then breaks down into quarterly goals. These are further divided into monthly KPIs by reversing through the pipeline: from revenue to awareness. These monthly KPIs are reviewed weekly in meetings with the leadership team and the KPI owners to ensure alignment and prompt action on emerging challenges and opportunities. 6. In a few sentences, what do you offer to whom? We provide a top-tier, developer-centric secrets management solution tailored for medium to large-scale companies. Our platform equips engineering teams with robust tools to store, manage, and orchestrate access to their secrets across various projects, teams, and environments, enhancing both security and operational efficiency. 7. What’s most exciting about your traction to date? The most thrilling aspect of our progress is our upward trajectory into larger markets. Our core mission is to secure the world's secrets, and it is particularly the larger enterprises that hold the highest density of sensitive information worth protecting. The recognition and adoption of our platform by these substantial institutions is not only a testament to the value we provide but also highlights the crucial need for our services in safeguarding their extensive digital assets. This shift is both validating and motivating as we continue to expand our reach and impact. 8. Where do you think your growth will be next year? While I am cautious about making speculative forecasts far in advance, if pressed for an estimate, I anticipate seeing our revenue grow by approximately 2 to 4x next year. 9. Tell us about your first paying customer and revenue expectations over the next year. Our very first paying customer was Tonebase! I had the opportunity to meet the founders through mutual friends and introduced them to our product with the Chipotle Sales Method. Their support was invaluable during our formative days—they provided consistent, high-quality feedback that was instrumental in shaping the foundational aspects of our product. Their early endorsement not only boosted our confidence but also set a strong precedent for the level of engagement we strive to maintain with all our customers. As for the upcoming year, we are optimistic about our revenue growth, aiming to build on this strong base and expand our reach further. 10. What’s your biggest threat? I love this question! As a security company, our most significant threat is undoubtedly the possibility of being hacked. We dedicate an immense amount of time, resources, and energy to ensure this does not happen, going above and beyond standard security practices. Despite these rigorous efforts, the risk remains—not just for us, but also for our customers. We believe in transparency and talk openly about the proactive steps we take to mitigate potential risks. Get Interviewed by HackerNoon via the Startup Founder Template 10 Questions with Founder and CEO Brian Vallelunga at Doppler via [ The Startup Founder Interview Template ]. Brian Vallelunga Doppler The Startup Founder Interview Template 1. What is your company in 2–5 words? Scalable secrets management for developers. 2. Why is now the time for your company to exist? Developers are increasingly dependent on numerous services, each requiring authentication through API keys, database URLs, and other sensitive secrets. Secure management of these secrets is crucial, not only to safeguard a company's reputation but more importantly, to protect the personal data of potentially millions of users from breaches. As the complexity of digital services grows, the need for robust, streamlined solutions to secure and manage these secrets becomes more pressing. Our company exists to address this vital need, ensuring that security keeps pace with innovation and protecting all stakeholders involved. 3. What do you love about your team, and why are you the ones to solve this problem? It's often said, but in our case, it genuinely applies—our team is truly unique. What sets us apart is not just the expertise or the skill each member brings, but their extraordinary passion for our mission. This passion isn't just about the work; it's about making a meaningful impact. This shared commitment has fostered one of the most cohesive and vibrant cultures I've ever had the pleasure of being part of. It's this combination of passion, culture, and talent that uniquely positions us to tackle such challenging problems effectively. We aren't just working; we're driven by a collective mission to innovate and lead in our field, ensuring we provide the best solutions. 4. If you weren’t building your startup, what would you be doing? If I weren't dedicated to building our startup, I would likely be involved in another venture within the hard tech space, particularly one that leverages artificial intelligence to accelerate research and development. AI has the potential to vastly improve efficiency and innovation across various industries, and I'm deeply interested in exploring how these technologies can solve complex problems and drive progress. 5. At the moment, how do you measure success? What are your metrics? We operate our company based on a well-defined funnel, where each stage is guided by specific input and output KPIs, each assigned to a dedicated owner. Teams responsible for a shared stage in the funnel—both input and output KPIs—are required to hold weekly meetings to continuously refine and improve their performance metrics. At the top of the funnel, we focus on awareness KPIs, such as website visits, while at the bottom, we measure success through financial-based KPIs like the Annual Recurring Revenue (ARR) from self-serve and managed sales. To set our financial objectives, we begin with annual revenue targets established by the board and CEO, which our finance team then breaks down into quarterly goals. These are further divided into monthly KPIs by reversing through the pipeline: from revenue to awareness. These monthly KPIs are reviewed weekly in meetings with the leadership team and the KPI owners to ensure alignment and prompt action on emerging challenges and opportunities. 6. In a few sentences, what do you offer to whom? We provide a top-tier, developer-centric secrets management solution tailored for medium to large-scale companies. Our platform equips engineering teams with robust tools to store, manage, and orchestrate access to their secrets across various projects, teams, and environments, enhancing both security and operational efficiency. 7. What’s most exciting about your traction to date? The most thrilling aspect of our progress is our upward trajectory into larger markets. Our core mission is to secure the world's secrets, and it is particularly the larger enterprises that hold the highest density of sensitive information worth protecting. The recognition and adoption of our platform by these substantial institutions is not only a testament to the value we provide but also highlights the crucial need for our services in safeguarding their extensive digital assets. This shift is both validating and motivating as we continue to expand our reach and impact. 8. Where do you think your growth will be next year? While I am cautious about making speculative forecasts far in advance, if pressed for an estimate, I anticipate seeing our revenue grow by approximately 2 to 4x next year. 9. Tell us about your first paying customer and revenue expectations over the next year. Our very first paying customer was Tonebase! I had the opportunity to meet the founders through mutual friends and introduced them to our product with the Chipotle Sales Method. Their support was invaluable during our formative days—they provided consistent, high-quality feedback that was instrumental in shaping the foundational aspects of our product. Their early endorsement not only boosted our confidence but also set a strong precedent for the level of engagement we strive to maintain with all our customers. As for the upcoming year, we are optimistic about our revenue growth, aiming to build on this strong base and expand our reach further. 10. What’s your biggest threat? I love this question! As a security company, our most significant threat is undoubtedly the possibility of being hacked. We dedicate an immense amount of time, resources, and energy to ensure this does not happen, going above and beyond standard security practices. Despite these rigorous efforts, the risk remains—not just for us, but also for our customers. We believe in transparency and talk openly about the proactive steps we take to mitigate potential risks. Get Interviewed by HackerNoon via the Startup Founder Template Get Interviewed by HackerNoon via the Startup Founder Template

Tech Founder Interview on Scalable Secrets Management for Developers and Scaling Yearly Revenue 2-4X

বিকাশকারীদের জন্য স্কেলেবল সিক্রেটস ম্যানেজমেন্ট এবং বার্ষিক আয় 2-4X স্কেল করার বিষয়ে টেক প্রতিষ্ঠাতা সাক্ষাৎকার

Interview mit Tech-Gründer zum Thema skalierbares Geheimnismanagement für Entwickler und Steigerung des Jahresumsatzes um das Zwei- bis Vierfache

Entrevista al fundador de tecnología sobre la gestión escalable de secretos para desarrolladores y el aumento de los ingresos anuales de 2 a 4 veces

Entretien avec le fondateur technique sur la gestion évolutive des secrets pour les développeurs et la multiplication par 2 à 4 des revenus annuels

डेवलपर्स के लिए स्केलेबल सीक्रेट्स मैनेजमेंट और वार्षिक राजस्व को 2-4X तक बढ़ाने पर टेक संस्थापक का साक्षात्कार

開発者向けのスケーラブルなシークレット管理と年間収益の 2 ～ 4 倍の拡大に関する技術創業者インタビュー

개발자를 위한 확장 가능한 비밀 관리 및 연간 수익 2~4배 확장에 대한 기술 창립자 인터뷰

Entrevista com o fundador da tecnologia sobre gerenciamento escalonável de segredos para desenvolvedores e aumento da receita anual de 2 a 4 vezes

Интервью с основателем технологии о масштабируемом управлении секретами для разработчиков и увеличении годового дохода в 2–4 раза

Geliştiriciler için Ölçeklenebilir Gizli Bilgi Yönetimi ve Yıllık Gelirin 2-4 Kat Arttırılması Konusunda Teknoloji Kurucusu Röportajı

Phỏng vấn người sáng lập công nghệ về quản lý bí mật có thể mở rộng dành cho nhà phát triển và mở rộng quy mô Doanh thu hàng năm gấp 2-4 lần

技术创始人访谈：面向开发人员的可扩展机密管理以及将年收入扩大 2-4 倍

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 5 Discussion The purpose of this study was to investigate the effects of a gamification element in a mathematical sciences course taught in flipped format. The study compared levels of engagement with pre-lecture quizzes before and after the gamification intervention was introduced. Comparisons were made across semesters (different courses, same students) and across academic years (same course, different students). Our study found that in both cases the level of engagement (measured by average percentage of students completing pre-lecture quizzes prior to the associated in-person sessions) was significantly higher when compared to a course where Level Up! was not implemented. Although engagement levels did drop over the course of each semester, they remained substantially higher for each week when compared to the previous semester or year. These results provide convincing evidence that introducing a gamification element to a course taught in flipped format increases students motivation and engagement to complete work outside of scheduled class time. When comparing engagement levels within students between two semesters, we found a general increase in engagement across both semesters but more interestingly found that some students with very low engagement levels in semester 1 went on the have some of the highest engagement levels in semester 2. For these students it seems that the gamification element drastically improved their motivation and engagement levels and is evidence that including a gamification element to a course may encourage engagement from students who would otherwise not engage. Our results also found that this increase in engagement could lead to an increase in performance with 82% of students receiving a grade of A, B or C in the semester 1 course of 2023-24 compared to 60% in 2022-23. Although we cannot say for sure what led to this increase in achievement and there are many factors which could influence this (such as cohort effect, different exam paper, etc.), an increase in engagement throughout a semester is compatible with an increase in performance since students are more likely to be distributing their learning over the course of a semester. This is especially beneficial for mathematical sciences courses where learning is often thought of as linear with each topic building on the previous one and poor understanding of early material can have a detrimental effect later in the course [Ireland and Mouthaan, 2020]. This study focuses on Year 1 undergraduate students who typically fall into a post-secondary education age range. The transition to university for students can be overwhelming, with changes in teaching delivery, workload and cognitive demands ([Blair, 2017]). Our results show that gamification can improve the learning experience, which may help with this transition, in line with observations by [Zaric et al., 2017]. This being said, [Kim and Castelli, 2021] looked at the effects of gamification across three age cohorts and found that university level participants found the least significant results in terms of positive interaction when compared to secondary school and adult learners, indicating that younger and older learners may have more interest in gamified factors. Currently, gamification is only present within the Year 1 modules of the undergraduate programme, and is not implemented within other modules on the programme. There is scope to explore the effects of gamification on a more mature cohort to observe if we see similar patterns of engagement. This study has considered student engagement levels in terms of completing pre-lecture quizzes, though it is important to note that we see meaningful engagement instead of cases where students are simply trying to ‘game’ the leaderboard. Our results are based solely on participation on formative reading quizzes where participation is identified through criteria being met, as this material was found to show that genuine participation took place. There are other activities in the course, such as textbook readings, which simply require a manual completion indicator and could be exposed to non-meaningful engagement. We feel that a reasonable performance on the quizzes indicates engagement with the other activities available (or indeed, of students independently finding their own resources for study). The results presented here are for a self-selected subset of students. However, course instructors were able to view results for the entire cohort of students to inform their future decision making on the learning and teaching approach for this course and others. The results presented here were found to be similar to the results for the entire cohort and the general conclusions made would not change if data on the entire cohort of students were considered. Although studies in gamification have reflected positive outcomes in terms of increased motivation and engagement in tasks, this teaching strategy has potential downsides. Effects such as increased competition between students ([Hakulinen et al., 2013]) and task evaluation difficulties, where certain tasks may not be suitable for a game based element or may have ways of being marked as complete without proper engagement in the learning materials ([Domínguez et al., 2013]) may become issues. The design of game based elements within a course must also be considered with a balance of reward and level of work and engagement required. Tasks which are overly complex can lead to a lack of engagement from learners ([Dong et al., 2012]). There is also potential for demotivation through the full gamification process, particulaly if students find themselves placed low on the points leaderboard. The competency levels of students may lead to negative consequences, including frustration and loss of self confidence ([Alomari et al., 2019]). A study by [Ding et al., 2018] showed that some students in a gamified learning activity may require more time to obtain the necessary understanding to progress through learning process. Engagement in gamification not only requires active participation from students, but time and attention from staff to implement the system effectively. From our experience using Level Up!, it takes time to set up the system. Several considerations have to be made at the set-up stage, including the allocation of experience points to certain tasks, and allocating points in such a way as to reflect the level of engagement required from the student. Levels were attainable for students which were split by weekly tasks, which requires manual intervention to tally the available points over a given week. For example, students could receive half points for completing certain tasks after a deadline which requires manual intervention from the course lecturer to reset the points awarded to a different total after each deadline. Though our study explored gamification using Level Up!, there are a host of other plug-ins which can be adapted in virtual learning environments such as Quizventure[4] and Block Game[5]. [Heilbrunn et al., 2017] provide a detailed review of tools for gamification analytics. Finally, we highlight the importance of using learning analytics, where possible, to access data from students on a course when measuring the success of a new teaching method or intervention. If the instructors of the two courses under consideration here had relied on feedback from students to inform teaching strategy - which was consistently positive even before the implementation of Level Up! - the true extent of the lack of engagement in the pre-lecture materials may never have been identified. This paper is available on arxiv under CC BY 4.0 DEED license. [4] https://github.com/xow/moodle-mod_quizgame [5] https://moodle.org/plugins/block_game Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 5 Discussion The purpose of this study was to investigate the effects of a gamification element in a mathematical sciences course taught in flipped format. The study compared levels of engagement with pre-lecture quizzes before and after the gamification intervention was introduced. Comparisons were made across semesters (different courses, same students) and across academic years (same course, different students). Our study found that in both cases the level of engagement (measured by average percentage of students completing pre-lecture quizzes prior to the associated in-person sessions) was significantly higher when compared to a course where Level Up! was not implemented. Although engagement levels did drop over the course of each semester, they remained substantially higher for each week when compared to the previous semester or year. These results provide convincing evidence that introducing a gamification element to a course taught in flipped format increases students motivation and engagement to complete work outside of scheduled class time. When comparing engagement levels within students between two semesters, we found a general increase in engagement across both semesters but more interestingly found that some students with very low engagement levels in semester 1 went on the have some of the highest engagement levels in semester 2. For these students it seems that the gamification element drastically improved their motivation and engagement levels and is evidence that including a gamification element to a course may encourage engagement from students who would otherwise not engage. Our results also found that this increase in engagement could lead to an increase in performance with 82% of students receiving a grade of A, B or C in the semester 1 course of 2023-24 compared to 60% in 2022-23. Although we cannot say for sure what led to this increase in achievement and there are many factors which could influence this (such as cohort effect, different exam paper, etc.), an increase in engagement throughout a semester is compatible with an increase in performance since students are more likely to be distributing their learning over the course of a semester. This is especially beneficial for mathematical sciences courses where learning is often thought of as linear with each topic building on the previous one and poor understanding of early material can have a detrimental effect later in the course [Ireland and Mouthaan, 2020]. This study focuses on Year 1 undergraduate students who typically fall into a post-secondary education age range. The transition to university for students can be overwhelming, with changes in teaching delivery, workload and cognitive demands ([Blair, 2017]). Our results show that gamification can improve the learning experience, which may help with this transition, in line with observations by [Zaric et al., 2017]. This being said, [Kim and Castelli, 2021] looked at the effects of gamification across three age cohorts and found that university level participants found the least significant results in terms of positive interaction when compared to secondary school and adult learners, indicating that younger and older learners may have more interest in gamified factors. Currently, gamification is only present within the Year 1 modules of the undergraduate programme, and is not implemented within other modules on the programme. There is scope to explore the effects of gamification on a more mature cohort to observe if we see similar patterns of engagement. This study has considered student engagement levels in terms of completing pre-lecture quizzes, though it is important to note that we see meaningful engagement instead of cases where students are simply trying to ‘game’ the leaderboard. Our results are based solely on participation on formative reading quizzes where participation is identified through criteria being met, as this material was found to show that genuine participation took place. There are other activities in the course, such as textbook readings, which simply require a manual completion indicator and could be exposed to non-meaningful engagement. We feel that a reasonable performance on the quizzes indicates engagement with the other activities available (or indeed, of students independently finding their own resources for study). The results presented here are for a self-selected subset of students. However, course instructors were able to view results for the entire cohort of students to inform their future decision making on the learning and teaching approach for this course and others. The results presented here were found to be similar to the results for the entire cohort and the general conclusions made would not change if data on the entire cohort of students were considered. Although studies in gamification have reflected positive outcomes in terms of increased motivation and engagement in tasks, this teaching strategy has potential downsides. Effects such as increased competition between students ([Hakulinen et al., 2013]) and task evaluation difficulties, where certain tasks may not be suitable for a game based element or may have ways of being marked as complete without proper engagement in the learning materials ([Domínguez et al., 2013]) may become issues. The design of game based elements within a course must also be considered with a balance of reward and level of work and engagement required. Tasks which are overly complex can lead to a lack of engagement from learners ([Dong et al., 2012]). There is also potential for demotivation through the full gamification process, particulaly if students find themselves placed low on the points leaderboard. The competency levels of students may lead to negative consequences, including frustration and loss of self confidence ([Alomari et al., 2019]). A study by [Ding et al., 2018] showed that some students in a gamified learning activity may require more time to obtain the necessary understanding to progress through learning process. Engagement in gamification not only requires active participation from students, but time and attention from staff to implement the system effectively. From our experience using Level Up!, it takes time to set up the system. Several considerations have to be made at the set-up stage, including the allocation of experience points to certain tasks, and allocating points in such a way as to reflect the level of engagement required from the student. Levels were attainable for students which were split by weekly tasks, which requires manual intervention to tally the available points over a given week. For example, students could receive half points for completing certain tasks after a deadline which requires manual intervention from the course lecturer to reset the points awarded to a different total after each deadline. Though our study explored gamification using Level Up!, there are a host of other plug-ins which can be adapted in virtual learning environments such as Quizventure[4] and Block Game[5]. [Heilbrunn et al., 2017] provide a detailed review of tools for gamification analytics. Finally, we highlight the importance of using learning analytics, where possible, to access data from students on a course when measuring the success of a new teaching method or intervention. If the instructors of the two courses under consideration here had relied on feedback from students to inform teaching strategy - which was consistently positive even before the implementation of Level Up! - the true extent of the lack of engagement in the pre-lecture materials may never have been identified. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [4] https://github.com/xow/moodle-mod_quizgame [5] https://moodle.org/plugins/block_game

Levelling Up Learning: Exploring the Impact of Gamification in Flipped Classrooms: Discussion

Levelling Up Learning: Exploring the Impact of Gamification in Flipped Classrooms: Findings

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 3 Motivation and intervention The pre-lecture activities were made available to students via the University of Glasgow’s virtual learning environment Moodle [2]. As such, it is straightforward for course instructors to check if students are engaging with these activities and crucially, given the flipped learning approach, whether they are engaging with them in advance of the in-person sessions. As discussed in Section 2.1, the University of Glasgow Learning Analytics Policy allows staff to undertake learning analytics activities which enable interventions to assist students learning. From the year in which the flipped learning approach was introduced (2020-21) staff on the course were therefore monitoring engagement in pre-lecture activities through activity logs on Moodle. For the purpose of this study, only engagement with the formative reading quizzes (pre-lecture quizzes) is measured, where activity completion can be indicated by certain conditions being met (e.g. whether a student has received a grade for a quiz), rather than students being able to manually mark completion (e.g. for link to reading from course textbook). This means that we cannot measure and compare engagement across all pre-lecture activities and we may be misrepresenting true engagement as some students may choose to read the textbook but not complete the reading quiz (or vice versa). However, this should give a more reliable estimate of meaningful engagement in activities since students have to engage somewhat with the activity (e.g. start and submit the quiz) rather than being able to manually mark that the activity is complete by checking a button. This does not fully guard against non-meaningful engagement (e.g. a student could begin the quiz, complete no questions, submit, and still be counted as having completed this activity) but performance in these activities (e.g. questions attempted and marks achieved) was also checked to understand the extent of meaningful engagement. Despite students indicating that they enjoyed the flipped learning approach through course evaluation forms, when looking at the data it was clear that engagement in the pre-lecture quizzes was extremely low. For example, the average completion of pre-lecture quizzes in advance of the active learning session taking place was 6.88% in semester 1 of 2022-23. This was of concern to course instructors given the in-class sessions involved spending less time presenting material and more time engaging in active learning activities as per the goal of flipped learning. This pattern was persistent over a number of years. Given that the introduction of the new courses coincided with the Covid-19 pandemic, it was also difficult to disentangle the lack of engagement with the flipped learning material, and the general lack of engagement HE institutions saw over the course of the pandemic ([Wester et al., 2021]). The academic year of 2022-23 saw a return to campus for the majority of teaching but the issue of low engagement in pre-lecture activities persisted. This is demonstrated in Figure 1 (a) which shows the cumulative mean percentage of students engaging in the pre-lecture quizzes for each week across an 11 week period in semester 1 of 2022-23. Each coloured line represents the cumulative average percentage of students who have engaged in the collection of quizzes associated with one week of teaching over the course of the semester (an average is shown here since each week may have a number of quizzes for students to complete). Of most interest is the level of engagement in the pre-lecture quizzes for the week in which the material is covered in class, which is represented by the first point for each weekly line in Figure 1 (a). These figures are also presented in column 1 of Table 1. This figure starts off low (around 27%) for week 1 and continues to drop for each week of material, before reaching a point around week 7 where less than 5% of students have engaged with the pre-lecture quizzes on the week in which the material is covered in class. Another interesting feature of Figure 1 (a) is the pattern of engagement for each collection of weekly quizzes over the semester. Although the engagement in the weekly material does increase slightly over the semester, at most only about half of the students in the class have ever engaged with the material at all on average for week 1, and after week 8 this drops to less than 25%. Interestingly, for all weeks an increase in average engagement can be seen between weeks 10 and 11 and we believe this to be related to exam preparation as the exam takes place immediately after the course finishes. These features are clearly of huge concern to the course instructors as they indicate that not only are students not engaging with the pre-lecture quizzes ‘on time’ to benefit from a flipped approach, the majority of students aren’t engaging with the materials at all across the entire duration of the course. In an attempt to tackle this low engagement the decision was made to add an element of gamification to the course which would encourage students to engage in the pre-lecture activities. This was implemented through the the Level Up! Moodle plug-in[3] which was introduced to the course in semester 2 of 2022-23. Level Up! allows students to gain points by participating in the course via Moodle. Points were awarded for the completion for a variety of course activities including the pre-lecture activities, tutorial questions, lab materials, and assessed quizzes and points were awarded based on completion only, i.e. the points system was endeavour-based and not merit-based. In an attempt to explicitly tackle engagement in the pre-lecture activities, full points were only awarded if the activity was completed before the associated lecture/tutorial/lab with half-points being awarded thereafter. Figure 2 shows how this plug-in is integrated within the course Moodle page with the left-hand image showing the Level Up! block that students can see on the main course page and the right-hand image showing the leaderboard view which shows students where they rank individually on an anonymous leaderboard throughout the semester. There is also an option to view where the students’ tutorial group rank in a team leaderboard. Level Up! was initially introduced in an attempt to increase engagement in pre-lecture activities in semester 2 of the 2022-23 academic year and, after initial success, introduced from semester 1 of the 2023-24 academic year. This allows us to make a number of comparisons which form our questions of interest: How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses.


How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. This paper is available on arxiv under CC BY 4.0 DEED license. [2] https://moodle.org/ [3] https://moodle.org/plugins/block_xp Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 3 Motivation and intervention The pre-lecture activities were made available to students via the University of Glasgow’s virtual learning environment Moodle [2]. As such, it is straightforward for course instructors to check if students are engaging with these activities and crucially, given the flipped learning approach, whether they are engaging with them in advance of the in-person sessions. As discussed in Section 2.1, the University of Glasgow Learning Analytics Policy allows staff to undertake learning analytics activities which enable interventions to assist students learning. From the year in which the flipped learning approach was introduced (2020-21) staff on the course were therefore monitoring engagement in pre-lecture activities through activity logs on Moodle. For the purpose of this study, only engagement with the formative reading quizzes (pre-lecture quizzes) is measured, where activity completion can be indicated by certain conditions being met (e.g. whether a student has received a grade for a quiz), rather than students being able to manually mark completion (e.g. for link to reading from course textbook). This means that we cannot measure and compare engagement across all pre-lecture activities and we may be misrepresenting true engagement as some students may choose to read the textbook but not complete the reading quiz (or vice versa). However, this should give a more reliable estimate of meaningful engagement in activities since students have to engage somewhat with the activity (e.g. start and submit the quiz) rather than being able to manually mark that the activity is complete by checking a button. This does not fully guard against non-meaningful engagement (e.g. a student could begin the quiz, complete no questions, submit, and still be counted as having completed this activity) but performance in these activities (e.g. questions attempted and marks achieved) was also checked to understand the extent of meaningful engagement. Despite students indicating that they enjoyed the flipped learning approach through course evaluation forms, when looking at the data it was clear that engagement in the pre-lecture quizzes was extremely low. For example, the average completion of pre-lecture quizzes in advance of the active learning session taking place was 6.88% in semester 1 of 2022-23. This was of concern to course instructors given the in-class sessions involved spending less time presenting material and more time engaging in active learning activities as per the goal of flipped learning. This pattern was persistent over a number of years. Given that the introduction of the new courses coincided with the Covid-19 pandemic, it was also difficult to disentangle the lack of engagement with the flipped learning material, and the general lack of engagement HE institutions saw over the course of the pandemic ([Wester et al., 2021]). The academic year of 2022-23 saw a return to campus for the majority of teaching but the issue of low engagement in pre-lecture activities persisted. This is demonstrated in Figure 1 (a) which shows the cumulative mean percentage of students engaging in the pre-lecture quizzes for each week across an 11 week period in semester 1 of 2022-23. Each coloured line represents the cumulative average percentage of students who have engaged in the collection of quizzes associated with one week of teaching over the course of the semester (an average is shown here since each week may have a number of quizzes for students to complete). Of most interest is the level of engagement in the pre-lecture quizzes for the week in which the material is covered in class, which is represented by the first point for each weekly line in Figure 1 (a). These figures are also presented in column 1 of Table 1. This figure starts off low (around 27%) for week 1 and continues to drop for each week of material, before reaching a point around week 7 where less than 5% of students have engaged with the pre-lecture quizzes on the week in which the material is covered in class. Another interesting feature of Figure 1 (a) is the pattern of engagement for each collection of weekly quizzes over the semester. Although the engagement in the weekly material does increase slightly over the semester, at most only about half of the students in the class have ever engaged with the material at all on average for week 1, and after week 8 this drops to less than 25%. Interestingly, for all weeks an increase in average engagement can be seen between weeks 10 and 11 and we believe this to be related to exam preparation as the exam takes place immediately after the course finishes. These features are clearly of huge concern to the course instructors as they indicate that not only are students not engaging with the pre-lecture quizzes ‘on time’ to benefit from a flipped approach, the majority of students aren’t engaging with the materials at all across the entire duration of the course. In an attempt to tackle this low engagement the decision was made to add an element of gamification to the course which would encourage students to engage in the pre-lecture activities. This was implemented through the the Level Up! Moodle plug-in[3] which was introduced to the course in semester 2 of 2022-23. Level Up! allows students to gain points by participating in the course via Moodle. Points were awarded for the completion for a variety of course activities including the pre-lecture activities, tutorial questions, lab materials, and assessed quizzes and points were awarded based on completion only, i.e. the points system was endeavour-based and not merit-based. In an attempt to explicitly tackle engagement in the pre-lecture activities, full points were only awarded if the activity was completed before the associated lecture/tutorial/lab with half-points being awarded thereafter. Figure 2 shows how this plug-in is integrated within the course Moodle page with the left-hand image showing the Level Up! block that students can see on the main course page and the right-hand image showing the leaderboard view which shows students where they rank individually on an anonymous leaderboard throughout the semester. There is also an option to view where the students’ tutorial group rank in a team leaderboard. Level Up! was initially introduced in an attempt to increase engagement in pre-lecture activities in semester 2 of the 2022-23 academic year and, after initial success, introduced from semester 1 of the 2023-24 academic year. This allows us to make a number of comparisons which form our questions of interest: How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [2] https://moodle.org/ [3] https://moodle.org/plugins/block_xp

Exploring the Impact of Gamification in Flipped Classrooms: Motivation and intervention

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 2 Course information At the University of Glasgow, the School of Mathematics and Statistics offer courses in introductory Statistics to undergraduate students across the University. These level 1 courses are typically taken in year 1 or 2 (pre-honours) of a student’s university career, which typically lasts 4 years. In year 1, single honours students will take courses in three subjects, one of which will be the subject they have chosen to specialise in. Any of these three courses can be considered in year 2, and students have the option to switch their chosen degree programme to any of these subjects conditional on meeting the requirements for progression. The purpose of the level 1 statistics courses is predominately to prepare students who are interested in studying statistics at higher levels and, given that most students will have little experience of statistics when they arrive at university, introduce students to the subject with the hope that students will choose to continue studying statistics in future years. There are two level 1 courses taught over two semesters (semester 1: Sep-Dec and semester 2: Jan-Mar) with final exams taking place at the end of each semester. Given the growing awareness of the importance of statistics across many disciplines, the courses have become increasingly popular over the past few years with 200+ students enrolled. To align with the University of Glasgow Learning and Teaching strategy, the courses were re-designed for 2020-21 with the aim of evolving the approach to learning and teaching towards student-centered, active learning. The approach taken for these courses was to introduce a flipped learning model, with a variety of online learning materials for students to complete before live teaching sessions. In-person sessions were then used to consolidate learning through a variety of interactive activities for students to complete individually or in small groups. For the purpose of this study we will call these online learning materials, ‘pre-lecture activities’. 2.1 Data The data for this study are taken from Moodle activity logs. These can be used to view all student interactions with Moodle, including engagement with the activities of interest for the courses under consideration here. Data such as these are becoming increasingly available and the use of data in the support of student learning is commonly referred to as learning analytics ([Leitner et al., 2017]). At this institution, staff may undertake learning analytics activities which enable interventions to assist students through the analysis of their own data. However, when such projects are intended for external dissemination they also require ethical approval. This study falls into the latter category and ethical approval was sought and granted in advance of this study taking place (application number 300220006). The research ethics associated with this study required students to consent to their data being used as part of this project. Consent was requested during live teaching sessions and was granted for n = 100 (out of a total of 257) students in 2022-23 and n = 77 (out of a total of 280) students in 2023-24. Unfortunately we were unable to obtain retrospective ethical approval to access data from 2020-21 (when flipped learning was introduced) and 2021-22 due to the opt-in nature of the study. We can therefore only provide results and comparisons for engagement using the 2022-23 and 2023-24 cohorts. Students attended three in-person sessions a week, and before each of these were asked to complete the following pre-lecture activities: • Textbook readings - students were asked to read a section of the course textbook in advance of the associated in-person session. The purpose of the readings was to introduce students to the course material. Completion of these can only be tracked through students manually marking completion via activity tracking on Moodle. • Formative reading quizzes - students were asked to complete a short quiz associated with after reading the corresponding textbook section. The purpose of the quizzes was for students to consolidate their learning and test their understanding of the course material in advance of the corresponding in-person session. Information on attempts for all quizzes is automatically stored in Moodle. It is therefore possible to determine which students have completed a quiz, when they started/finished the quiz, and their performance on the quiz. We can therefore track engagement in the quizzes and compare this with different cohorts as well as monitoring performance to inform the active learning sessions. These will be referred to as pre-lecture quizzes. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 2 Course information At the University of Glasgow, the School of Mathematics and Statistics offer courses in introductory Statistics to undergraduate students across the University. These level 1 courses are typically taken in year 1 or 2 (pre-honours) of a student’s university career, which typically lasts 4 years. In year 1, single honours students will take courses in three subjects, one of which will be the subject they have chosen to specialise in. Any of these three courses can be considered in year 2, and students have the option to switch their chosen degree programme to any of these subjects conditional on meeting the requirements for progression. The purpose of the level 1 statistics courses is predominately to prepare students who are interested in studying statistics at higher levels and, given that most students will have little experience of statistics when they arrive at university, introduce students to the subject with the hope that students will choose to continue studying statistics in future years. There are two level 1 courses taught over two semesters (semester 1: Sep-Dec and semester 2: Jan-Mar) with final exams taking place at the end of each semester. Given the growing awareness of the importance of statistics across many disciplines, the courses have become increasingly popular over the past few years with 200+ students enrolled. To align with the University of Glasgow Learning and Teaching strategy, the courses were re-designed for 2020-21 with the aim of evolving the approach to learning and teaching towards student-centered, active learning. The approach taken for these courses was to introduce a flipped learning model, with a variety of online learning materials for students to complete before live teaching sessions. In-person sessions were then used to consolidate learning through a variety of interactive activities for students to complete individually or in small groups. For the purpose of this study we will call these online learning materials, ‘pre-lecture activities’. 2.1 Data The data for this study are taken from Moodle activity logs. These can be used to view all student interactions with Moodle, including engagement with the activities of interest for the courses under consideration here. Data such as these are becoming increasingly available and the use of data in the support of student learning is commonly referred to as learning analytics ([Leitner et al., 2017]). At this institution, staff may undertake learning analytics activities which enable interventions to assist students through the analysis of their own data. However, when such projects are intended for external dissemination they also require ethical approval. This study falls into the latter category and ethical approval was sought and granted in advance of this study taking place (application number 300220006). The research ethics associated with this study required students to consent to their data being used as part of this project. Consent was requested during live teaching sessions and was granted for n = 100 (out of a total of 257) students in 2022-23 and n = 77 (out of a total of 280) students in 2023-24. Unfortunately we were unable to obtain retrospective ethical approval to access data from 2020-21 (when flipped learning was introduced) and 2021-22 due to the opt-in nature of the study. We can therefore only provide results and comparisons for engagement using the 2022-23 and 2023-24 cohorts. Students attended three in-person sessions a week, and before each of these were asked to complete the following pre-lecture activities: • Textbook readings - students were asked to read a section of the course textbook in advance of the associated in-person session. The purpose of the readings was to introduce students to the course material. Completion of these can only be tracked through students manually marking completion via activity tracking on Moodle. • Textbook readings • Formative reading quizzes - students were asked to complete a short quiz associated with after reading the corresponding textbook section. The purpose of the quizzes was for students to consolidate their learning and test their understanding of the course material in advance of the corresponding in-person session. Information on attempts for all quizzes is automatically stored in Moodle. It is therefore possible to determine which students have completed a quiz, when they started/finished the quiz, and their performance on the quiz. We can therefore track engagement in the quizzes and compare this with different cohorts as well as monitoring performance to inform the active learning sessions. These will be referred to as pre-lecture quizzes. • Formative reading quizzes This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Levelling Up Learning:Exploring the Impact of Gamification in Flipped Classrooms: Course information

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Conclusion Our analysis shows that while it may well be technically possible to design a search engine that manages to balance privacy-preservation with the promotion of high IQ material, this is currently not the case. The current relationship between privacy-preserving design features of search engines and the IQ of the results they return is inverse (although not proportionally). In instances where this can have a negative impact on public health, as in the example we have provided of the promotion of anti-vaccine misinformation, not intervening to alter the design of the algorithm even if this means sacrificing some degree of user privacy - can lead to severe harm for a large population of user and is, therefore, unethical. Designing a search engine that is privacy savvy and avoids issues with filter bubbles that can result from user-tracking may be a good thing, like designing a car with an engine that does not pollute and is inexpensive to run, and designers should seek to balance the different aspects of search engine design highlighted in Figure 3. However, if the brakes in an environmentally-designed car do not work, the car is unsafe and this negates the positive ethical decisions made by the designers. In a car this is a highly unlikely design outcome, as a car has to undergo several rounds of testing by regulatory agencies before being allowed on the market. This is not yet the case for search engines, which are only regulated from the perspective of data protection – which is primarily interpreted as data security rather than data privacy. Our study suggests that this is necessary but insufficient, and instead mechanisms should be developed to test search engines from the perspective of IQ, (particularly for YMYL webpages) before they can be deemed trustworthy providers of public health information. References 1. World Health Organization. Ten threats to global health in 2019. 2019. https://www.who.int/emergencies/ten-threats-to-global-health-in-2019 (archived at https://web.archive.org/web/20190627111040/https://www.who.int/emergencies/ten-threats-toglobal-health-in-2019) (accessed 27/06/2019 2. Wakefield AJ, Murch SH, Anthony A, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet 1998; 351(9103): 637-41. 3. Stahl JP, Cohen R, Denis F, et al. The impact of the web and social networks on vaccination. New challenges and opportunities offered to fight against vaccine hesitancy. Med Mal Infect 2016; 46(3): 117-22. 4. Martin GP, Sutton E, Willars J, Dixon-Woods M. Frameworks for change in healthcare organisations: a formative evaluation of the NHS Change Model. Health Serv Manage Res 2013; 26(2-3): 65-75. 5. Illari P, Floridi L. Information quality, data and philosophy. The Philosophy of Information Quality: Springer; 2014: 5-23. 6. Al-Jefri M, Evans R, Uchyigit G, Ghezzi P. What is health information quality? Ethical dimension and perception by users. Frontiers in medicine 2018; 5. 7. Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res 2006; 8(4): e27. 8. Chen X, Hay JL, Waters EA, et al. Health Literacy and Use and Trust in Health Information. J Health Commun 2018; 23(8): 724-34. 9. Paige SR, Krieger JL, Stellefson ML. The Influence of eHealth Literacy on Perceived Trust in Online Health Communication Channels and Sources. J Health Commun 2017; 22(1): 53-65. 10. Kim J, Park HA. Development of a health information technology acceptance model using consumers' health behavior intention. J Med Internet Res 2012; 14(5): e133. 11. Borah P, Xiao X. The Importance of 'Likes': The Interplay of Message Framing, Source, and Social Endorsement on Credibility Perceptions of Health Information on Facebook. J Health Commun 2018; 23(4): 399-411. 12. Huo C, Zhang M, Ma F. Factors influencing people’s health knowledge adoption in social media: the mediating effect of trust and the moderating effect of health threat. Library Hi Tech 2018; 36(1): 129-51. 13. Chiou L, Tucker C. Search engines and data retention: Implications for privacy and antitrust: National Bureau of Economic Research, 2017. 14. Esteve A. The business of personal data: Google, Facebook, and privacy issues in the EU and the USA. International Data Privacy Law 2017; 7(1): 36-47. 15. Floridi L, Taddeo M. What is data ethics? : The Royal Society; 2016. 16. Moskvitch K. Amazon’s Alexa deal with the NHS can help patients but at what cost? Wired, 2019. (accessed 8/11/2019). 17. Arif N, Al-Jefri M, Bizzi IH, et al. Fake News or Weak Science? Visibility and Characterization of Antivaccine Webpages Returned by Google in Different Languages and Countries. Front Immunol 2018; 9: 1215. 18. Maki A, Evans R, Ghezzi P. Bad news. Analysis of the quality of information on influenza prevention returned by Google in English and Italian. Frontiers in Immunology 2015; 6:616. doi: 10.3389/fimmu.2015.00616. 19. Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. Hoboken, NJ, USA: John Wiley & Sons; 2013. 20. Economist T. Seek and you shall find. Google rewards reputable reporting, not left-wing politics. The Economist 2019; 431(9146): 85. 21. Taddeo M, Floridi L. The Debate on the Moral Responsibilities of Online Service Providers. Science and engineering ethics 2015: 1-29. 22. Mittelstadt BD, Allo P, Taddeo M, Wachter S, Floridi L. The ethics of algorithms: Mapping the debate. Big Data & Society 2016; 3(2): 2053951716679679. 23. Association for Computing Machinery. ACM Code of Ethics and Professional Conduct. 2018. https://www.acm.org/code-of-ethics (archived at https://web.archive.org/web/20190627114921/https://www.acm.org/code-of-ethics) (accessed 27/06/2019 2019). 24. Tufekci Z. Algorithmic harms beyond Facebook and Google: Emergent challenges of computational agency. Colo Tech LJ 2015; 13: 203. 25. Hasnas J. Teaching business ethics: The principles approach. Journal of Business Ethics Education 2013; 10: 275-304. 26. Macdonald C, Gavura S. Alternative Medicine and the Ethics Of Commerce. Bioethics 2016; 30(2): 77-84. 27. Taddeo M, Floridi L. The Debate on the Moral Responsibilities of Online Service Providers. Sci Eng Ethics 2016; 22(6): 1575-603. 28. Chigwedere P, Seage GR, 3rd, Gruskin S, Lee TH, Essex M. Estimating the lost benefits of antiretroviral drug use in South Africa. J Acquir Immune Defic Syndr 2008; 49(4): 410-5. 29. Ouyang Y. Student's death highlights gaps in China's health regulations. Lancet Oncol 2016; 17(6): 709. 30. Browne M, Thomson P, Rockloff MJ, Pennycook G. Going against the Herd: Psychological and Cultural Factors Underlying the 'Vaccination Confidence Gap'. PLoS One 2015; 10(9): e0132562. 31. Bean SJ. Emerging and continuing trends in vaccine opposition website content. Vaccine 2011; 29(10): 1874-80.

Online Information of Vaccines: Conclusion and References

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 5 IMPLEMENTATION ConstitutionMaker is a web application and utilizes an LLM [3] that is promptable in the same way as GPT-3 [4] or PaLM [5]. In the following section, we go through the implementation of ConstitutionMaker’s key features. 5.1 Facilitating the Conversation To generate the chatbot’s response, ConstitutionMaker builds a dialogue prompt (Figure 3A) behind the scenes. The dialogue prompt consists of (1) a description of the bot’s capabilities, entered by the user (Figure 1A), (2) the current set of principles, and (3) the conversation history, ending with the user’s latest input. The prompt then generates the bot’s next response, for which we choose the top-3 completions outputted by the LLM to display to users (Figure 3B). When the conversation is restarted or rewound, the conversation history within the dialogue prompt is modified; in the case of restarting, the entire history is deleted, whereas for rewinding, everything after the rewind point is deleted. And finally, if the conversation gets too long for the prompt context window, we remove the oldest conversational turns until it fits. 5.2 Three Principle Elicitation Features All three principle elicitation features output a principle that is then incorporated back into the dialogue prompt (Figure 3A) to influence future conversational turns. Giving kudos and critiquing a bot’s response consist of a similar process. For both, the selected bot output is fed into a few-shot prompt that generates rationales, either positive (Figure 3C) or negative (Figure 3D). The user’s selected rationale (or their own written rationale) is then sent to a few-shot prompt that converts this rationale into a principle (Figure 3F and 3G). This few-shot prompt leverages the conversation history to create a specific, conditional principle. For example, for MusicBot, if the critique is “The bot did not ask questions about the user’s preferences,” a specific, conditional principle might be “Prior to giving a music recommendation, ask the user what genres or artists they currently listen to.” Next, for critiques, after the principle is inserted into the dialogue prompt, new outputs are generated to show to the user (Figure 3G). Finally, for rewriting the bot’s response, we leverage a chain-of-thought [38] style prompt that first generates a “thought,” which reasons how the original and rewritten outputs are different from each other, and then generates a specific principle based on that reasoning. Constructing the prompt with a “thought” portion led to principles that captured the difference between the two outputs better than our earlier versions without it. [3] anonymized for peer review [4] https://openai.com/api/ [5] https://developers.generativeai.google/ This paper is under CC 4.0 license. available on arxiv

How ConstitutionMaker Utilizes LLMs for Chatbot Behavior Crafting

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 9 CONCLUSION This paper presents ConstitutionMaker, a tool for interactively refining LLM outputs by converting users’ intuitive feedback into principles. ConstitutionMaker’s design is informed by a formative study, where we also collected and classified the types of principles users wanted to write. ConstitutionMaker incorporates three principle-elicitation features: kudos, critique, and rewrite. In a user study with 14 industry professionals, participants felt that ConstitutionMaker helped them (1) write principles that effectively guided the chatbot, (2) convert their feedback into principles more easily, and (3) write principles more efficiently, with (4) less mental demand than the baseline. This was due to ConstitutionMaker supporting their thought processes, including helping them to: identify ways to improve the bot’s responses, convert their intuition into verbal feedback, and phrase their feedback as specific principles. There are many avenues of future work, including supporting users in iterating on and clarifying their principles, organizing larger sets of principles and supporting multiple writers, and helping users test chatbots across multiple user journeys. Together, these findings inform future tools that support interactively customizing LLM outputs. REFERENCES [1] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy TelleenLawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073 [cs.CL] [2] Svetlin Bostandjiev, John O’Donovan, and Tobias Höllerer. 2012. TasteWeights: A Visual Interactive Hybrid Recommender System. In Proceedings of the Sixth ACM Conference on Recommender Systems (Dublin, Ireland) (RecSys ’12). Association for Computing Machinery, New York, NY, USA, 35–42. https://doi.org/10.1145/ 2365952.2365964 [3] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST). https: //doi.org/10.1145/3586183.3606725 arXiv:2304.09337 [cs.HC] [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf [5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv:2303.12712 [cs.CL] [6] Ernie Chang, Xiaoyu Shen, Hui-Syuan Yeh, and Vera Demberg. 2021. On Training Instance Selection for Few-Shot Neural Text Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Association for Computational Linguistics, Online, 8–13. https: //doi.org/10.18653/v1/2021.acl-short.2 [7] Qiaochu Chen, Xinyu Wang, Xi Ye, Greg Durrett, and Isil Dillig. 2020. MultiModal Synthesis of Regular Expressions. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation (London, UK) (PLDI 2020). Association for Computing Machinery, New York, NY, USA, 487–502. https://doi.org/10.1145/3385412.3385988 [8] Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for Science Writing Using Language Models. In Proceedings of the 2022 ACM Designing Interactive Systems Conference (Virtual Event, Australia) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 1002–1019. https: //doi.org/10.1145/3532106.3533533 [9] Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183. https://doi.org/10.1016/S0166- 4115(08)62386-9 [10] Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Finetuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 328–339. https://doi.org/ 10.18653/v1/P18-1031 [11] Jiyou Jia. 2009. CSIEC: A computer assisted English learning chatbot based on textual knowledge and reasoning. Knowledge-Based Systems 22, 4 (2009), 249–255. https://doi.org/10.1016/j.knosys.2008.09.001 Artificial Intelligence (AI) in Blended Learning. [12] Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. PromptMaker: Prompt-Based Prototyping with Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 35, 8 pages. https://doi.org/10.1145/3491101.3503564 [13] Ellen Jiang, Edwin Toh, Alejandra Molina, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2021. GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor. In Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21 Adjunct). Association for Computing Machinery, New York, NY, USA, 145–147. https://doi.org/10.1145/3474349.3480209 [14] Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 386, 19 pages. https://doi.org/10.1145/3491102.3501870 [15] Daniel Kahneman. 2011. Thinking, fast and slow. macmillan. [16] Tae Soo Kim, Yoonjoo Lee, Minsuk Chang, and Juho Kim. 2023. Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. ACM. 1–18. [17] Johannes Kunkel, Benedikt Loepp, and Jürgen Ziegler. 2017. A 3D Item Space Visualization for Presenting and Manipulating User Preferences in Collaborative Filtering. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (Limassol, Cyprus) (IUI ’17). Association for Computing Machinery, New York, NY, USA, 3–15. https://doi.org/10.1145/3025171.3025189 [18] Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. New England Journal of Medicine 388, 13 (2023), 1233–1239. https://doi.org/10.1056/NEJMsr2214184 arXiv:https://doi.org/10.1056/NEJMsr2214184 PMID: 36988602. [19] Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 3045–3059. https: //doi.org/10.18653/v1/2021.emnlp-main.243 [20] Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and CodeGenerating Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 598, 31 pages. https: //doi.org/10.1145/3544548.3580817 [21] Vivian Liu, Han Qiao, and Lydia Chilton. 2022. Opal: Multimodal Image Generation for News Illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 73, 17 pages. https://doi.org/10.1145/3526113.3545621 [22] Vivian Liu, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2023. 3DALLE: Integrating Text-to-Image AI in 3D Design Workflows. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (Pittsburgh, PA, USA) (DIS ’23). Association for Computing Machinery, New York, NY, USA, 1955–1977. https://doi.org/10.1145/3563657.3596098 [23] Aditi Mishra, Utkarsh Soni, Anjana Arunkumar, Jinbin Huang, Bum Chul Kwon, and Chris Bryan. 2023. PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models. arXiv:2304.01964 [cs.HC] [24] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442 [cs.HC] [25] Savvas Petridis, Nediyana Daskalova, Sarah Mennicken, Samuel F Way, Paul Lamere, and Jennifer Thom. 2022. TastePaths: Enabling Deeper Exploration and Understanding of Personal Preferences in Recommender Systems. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI ’22). Association for Computing Machinery, New York, NY, USA, 120–133. https: //doi.org/10.1145/3490099.3511156 [26] Savvas Petridis, Nicholas Diakopoulos, Kevin Crowston, Mark Hansen, Keren Henderson, Stan Jastrzebski, Jeffrey V Nickerson, and Lydia B Chilton. 2023. AngleKindling: Supporting Journalistic Angle Ideation with Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 225, 16 pages. https://doi.org/10.1145/3544548. 3580907 [27] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic Prompt Optimization with "Gradient Descent" and Beam Search. arXiv:2305.03495 [cs.CL] [28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9. [29] Kiran Ramesh, Surya Ravishankaran, Abhishek Joshi, and K. Chandrasekaran. 2017. A Survey of Design Techniques for Conversational Agents. In Information, Communication and Computing Technology, Saroj Kaushik, Daya Gupta, Latika Kharb, and Deepak Chahal (Eds.). Springer Singapore, Singapore, 336–350. [30] Emily Reif, Minsuk Kahng, and Savvas Petridis. 2023. Visualizing Linguistic Diversity of Text Datasets Synthesized by Large Language Models. arXiv:2305.11364 [cs.CL] [31] Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 314, 7 pages. https://doi.org/10.1145/3411763.3451760 [32] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 4222–4235. https://doi.org/10.18653/v1/2020. emnlp-main.346 [33] Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M. Rush. 2023. Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models. IEEE Transactions on Visualization and Computer Graphics 29, 1 (2023), 1146–1156. https://doi.org/10.1109/TVCG.2022.3209479 [34] Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2022. Selective Annotation Makes Language Models Better Few-Shot Learners. arXiv:2209.01975 [cs.CL] [35] Gust Verbruggen, Vu Le, and Sumit Gulwani. 2021. Semantic Programming by Example with Pre-Trained Models. Proc. ACM Program. Lang. 5, OOPSLA, Article 100 (oct 2021), 25 pages. https://doi.org/10.1145/3485477 [36] Sitong Wang, Savvas Petridis, Taeahn Kwon, Xiaojuan Ma, and Lydia B Chilton. 2023. PopBlends: Strategies for Conceptual Blending with Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 435, 19 pages. https://doi.org/10.1145/3544548. [37] Yunlong Wang, Shuyuan Shen, and Brian Y Lim. 2023. RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 22, 29 pages. https://doi.org/10.1145/3544548.3581402 [38] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 24824–24837. https://proceedings.neurips.cc/paper_files/paper/2022/file/ 9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf [39] Sherry Wu, Hua Shen, Daniel S Weld, Jeffrey Heer, and Marco Tulio Ribeiro. 2023. ScatterShot: Interactive In-Context Example Curation for Text Transformation. In Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23). Association for Computing Machinery, New York, NY, USA, 353–367. https://doi.org/10.1145/3581641.3584059 [40] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 359, 10 pages. https://doi.org/10.1145/3491101.3519729 [41] Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 385, 22 pages. https: //doi.org/10.1145/3491102.3517582 [42] Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential Matching Network: A New Architecture for Multi-turn Response Selection in RetrievalBased Chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 496–505. https://doi.org/10.18653/v1/P17-1046 [43] Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A New Chatbot for Customer Service on Social Media. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3506–3510. https://doi.org/10.1145/3025453.3025496 [44] Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI ’22). Association for Computing Machinery, New York, NY, USA, 841–852. https://doi.org/10.1145/3490099.3511105 [45] J.D. Zamfirescu-Pereira, Heather Wei, Amy Xiao, Kitty Gu, Grace Jung, Matthew G Lee, Bjoern Hartmann, and Qian Yang. 2023. Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (Pittsburgh, PA, USA) (DIS ’23). Association for Computing Machinery, New York, NY, USA, 2206–2220. https://doi.org/10.1145/3563657.3596138 [46] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https://doi.org/10.1145/ 3544548.3581388 [47] Tianyi Zhang, London Lowmanstone, Xinyu Wang, and Elena L. Glassman. 2020. Interactive Program Synthesis by Augmented Examples. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 627–648. https://doi.org/10.1145/3379337.3415900 [48] Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory. arXiv:1704.01074 [cs.CL] This paper is under CC 4.0 license. available on arxiv

Interactively Critiquing LLMs by Converting Feedback into Principle— Conclusion and References

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 8 DISCUSSION 8.1 Supporting Users in Clarifying and Iterating Over Principles Finding the right granularity for a principle was sometimes challenging for participants; they created under-specified principles that did not impact the conversation, as well as over-specified principles that applied too strictly to certain portions of the conversation. One way to support users in finding the right granularity could be generating questions to help them reflect on their principle. For example, if an abstract principle is written (e.g., “Ask follow up questions to understand the users preferences”), an LLM prompt can be used to pose clarifying questions, such as, “What kind of follow up questions should be asked?” or, “Should I do anything differently, depending on the user’s answers?” Users could then answer these questions the best they can and then the principle could be updated automatically. Alternatively, another way to help users reflect on their principles might be to engage in a side conversation with a chatbot to help clarify them. This chatbot could pose similar questions as those suggested above, but it might also provide examples of chat snippets that adhere to or violate the principle. As they converse, the chatbot might continue to pose clarifying questions, while the principle is updated on the fly. Thus, future work could examine supporting users in interactively reflecting upon and clarifying their principles. 8.2 Organizing Principles and Supporting Multiple Principle Writers As participants accumulated principles, it was increasingly likely that there was a conflict between two of them, and it was harder for them to get an overview of how their principles were affecting the conversation. One way to prevent potential principle conflicts is to leverage an LLM to conduct a pairwise comparison of principles to assess if any two are at odds, and then suggest a solution. This kind of conflict resolution, while useful for a single principle writer, would be crucial in cases when multiple individuals are writing principles to improve a model. Multiple principle writers would be useful for gathering feedback to improve the overall performance of the model, but with many generated principles, it is increasingly important to understand how they might impact the conversation. Perhaps a better way to prevent conflicts is to organize principles in a way that summarizes their impact on the conversation. Principles are small bits of computation; there are conditions when they are applicable, and depending on those conditions, the bot’s behavior might branch in separate directions. One way to organize principles is to construct a state diagram, which illustrates the potential set of supported user flows and bot responses. With this overview, users could be made better aware of the overall impact of their principles, and they could then easily revise it to prevent conflicts. Therefore, another rich vein of future work is developing mechanisms to resolve conflicts in larger sets of principles, as well as organizing them into an easily digestible overview. 8.3 Automatically Ideating Multiple User Journeys to Support Principle Writing To test the chatbot and identify instances where the model could be improved, participants employed a strategy where they went through different user journeys with the chatbot. Often their choice of user journeys were biased toward what they were interested in, or they only tested the most common journeys. One way to enable more robust testing of chatbots could be to generate potential user personas and journeys to inspire principle writers. Going further, these generated user personas could then be used simulate conversations with the chatbot being tested [24]. For example, for VacationBot, one user persona might be a parent looking for nearby, family friendly vacations. A dialogue prompt could be generated for this persona and then VacationBot and this test persona could converse for a predefined set of conversational turns. Afterwards, users could inspect the conversation, and edit or critique VacationBot’s responses to generate principles. This kind of workflow could sidestep the challenge of repeatedly shifting from an end-user’s perspective to a bot-designer’s perspective, which exists in the current workflow. At the same time, users would be able to evaluate fuller conversational arcs, as opposed to single conversational turns. Thus, another line of future work is supporting users in exploring diverse user journeys with their chatbot, as well as exploring workflows that require less perspective switching. 8.4 Limitations and Future Work A set of well-written principles is often not enough to robustly steer an LLM’s outputs. As more principles are written, an LLM might “forget” to apply older principles [45]. This work focuses on helping participants convert their intuitive feedback into clear principles, and we illustrate that the principle-elicitation features help with that process. However, in line with the original Constitutional AI workflow [1], future work can focus on using these principles to generate a fine-tuning dataset, so that the model robustly follows them. Next, while we selected chatbots for two very common use cases for the study (vacation and food), participants might not have been very knowledgeable or opinionated in these areas. Future work can explore how these principle-elicitation features help users when writing principles for chatbot use cases that they are experts in. That being said, it was necessary to choose two chatbot use cases for the study to enable a fair comparison across the two conditions. This paper is under CC 4.0 license. available on arxiv

The Future of Chatbot Customization

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 7 FINDINGS Quantitative Findings. From the exit interviews, 12 of 14 participants preferred ConstitutionMaker to the baseline version. The results from the questionnaire are summarized in Figure 4. We found that ConstitutionMaker was perceived to be more helpful for writing rules that effectively guided the bot (Z = 8, p = .007), scoring on average 5.79 (𝜎 = 1.01) whereas the baseline scored 4.0 (1.51). When participants rewound parts of the conversation, they thought the ConstitutionMaker principles were followed by the bot better. Next, participants felt it was significantly easier (Z = 10.5, p = .006) to convert their feedback into principles with ConstitutionMaker (𝜇 = 5.86, 𝜎 = 0.91) than with the baseline (𝜇 = 3.93, 𝜎 = 1.39). The automatic conversion of kudos and critiques into principles eased the process of converting intuitive feedback into clear criteria for the bot to follow. Participants perceived that they were significantly more efficient (Z = 5, p = .004) writing rules with ConstitutionMaker (𝜇 = 5.86, 𝜎 = 1.51) than with the baseline (𝜇 = 3.64, 𝜎 = 1.34). Finally, participants also felt that writing principles with ConstitutionMaker (𝜇 = 3.0, 𝜎 = 1.41) required significantly less mental demand (Z = 1.5, p = .002) than with the baseline (𝜇 = 5.21, 𝜎 =1.78). There was no statistically significant difference for diversity (Z = 19, p = .06); participants felt that they exercised their creativity and wrote relatively diverse principles in the baseline. Next, regarding the feature usage metrics, participants wrote significantly more principles (t(13)=4.73, p < .001) with ConstitutionMaker than with the baseline; participants wrote on average 6.78 (𝜎 = 2.11) principles per chatbot with ConstitutionMaker and 4.42 (𝜎 = 1.24) principles with the baseline. Of the 95 principles written in the ConstitutionMaker condition, 40 (42.1%) came from kudos, where 37 were selected from the generate rationales and 3 were written; 28 (29.5%) came from critique, where 8 were selected and 20 were written; 13 (13.6%) came from the rewrite features; 14 (14.7%) were manually written. Participants found rewriting a bit cumbersome and preferred the less intensive workflow of describing what they liked or did not like to generate principles. In the following sections, we provide further context to these findings. 7.1 Participants’ Workflows for Writing Principles The two conditions led to quite different workflows for writing principles. In the ConstitutionMaker condition, participants commonly scanned the three candidate outputs from the chatbot, identified the output they liked the most, and then gave kudos to that output if they thought it had a quality that was not currently reflected in their principles. For example, while P1 was working on FoodBot, he was asking for easy-to-make vegetarian dishes, and in one of the bot’s candidate outputs, each suggested dish had a short description and explanation on why it was easy. He appreciated this, skimmed the kudos, and then selected one that conveyed this positive feature of the bot’s output. However, if participants disliked all of the responses, they would then switch to critiquing one of the outputs. Accordingly, this kudos-first workflow helps to explain how most of the principles participants wrote came from that principle elicitation feature. Meanwhile, in the baseline condition, participants generally wrote principles when the bot deviated quite a bit (in a negative way) from what they expected. P8 explained, “Here it feels like what I more naturally do is write corrective rules, to guardrail anything that goes weird...If it’s already doing the right thing it doesn’t need a rule from me. I wouldn’t feel the need to write those.” In the baseline condition, participants only see one candidate output from the chatbot, and this might deemphasize the stochastic nature of the chatbot’s outputs. As a result, when participants are okay with a response, they could feel that they do not need to write a principle to encourage that kind of response further. Overall, with the baseline, participants predominantly wrote principles to steer the LLM from less optimal behavior, while with ConstitutionMaker, participants mostly used kudos the most to encourage behavior they liked. 7.2 ConstitutionMaker Supported Users’ Thought Processes In the following section, we discuss how ConstitutionMaker supported participants’ thought process from (1) forming an intuition on ways the chatbot could be improved, (2) expressing this intuition as feedback, and (3) converting this feedback into a specific and clear principle. As P5 was using the baseline after using ConstitutionMaker, she explained how she wished she could see multiple outputs again: “Sometimes, I don’t know what I’m missing [in the baseline]. I’m thinking of the Denali hiking example [which occurred when she wrote principles for VacationBot with ConstitutionMaker]. Two of the responses didn’t mention that Denali was good for young children. But one did, and I was able to pull that out as a positive principle.” While she was writing principles for VacationBot with ConstitutionMaker, P5 started off the conversation saying she was looking for suggestions for her family, which included two young children. As the conversation progressed and a general location was established, P5 then asked for hiking recommendations, for which the bot gave some, but only one of its responses highlighted that the hikes it was recommending were good for young children. P5 gave kudos to that response and created the following principle: “Consider information previously inputted by the user when providing recommendations.” Ultimately, it can be hard to form opinions on responses without getting exposed to alternatives, so by providing multiple chatbot outputs, ConstitutionMaker supported participants in forming an intuition on how the model might be steered. 7.2.1 Multiple chatbot outputs helped participants form an intuition on how the model could be steered. . Upon seeing a candidate response from the chatbot, participants could intuitively tell if they liked or disliked it, but struggled to articulate their thoughts. The automatically generated kudos and critiques helped participants recognize and formulate this feedback. For example, while working on FoodBot with ConstitutionMaker, P9 asked the bot to identify the pizzeria with the best thin crust from a list of restaurants provided in a prior turn. The bot responded with, “Pizzaiolo has the best thin crust pies.” P9 knew he did not like the response, so he went to critique it and selected the following generated option: “This response is bad because it does not provide any information about the other pizza places that the user asked about.” The following principle was generated: “If the user asks about a specific attribute of a list of items, provide information about all of the items in the list that have that attribute,” which then produced a set of revised responses that compared the qualities of each pizzeria’s crusts. Reflecting on this process, P9 stated, “I didn’t like that last answer [from FoodBot], but I didn’t have a concrete reason why yet...I didn’t really know how to put it into words yet...but reading the suggestions gave me at least one of many potential reasons on why I didn’t like the response.” Thus, ConstitutionMaker helped participants transition from fast thinking [15], that is, their intuitive and unconscious responses to the bot’s outputs, to slow thinking, a more deliberate, conscious formulation of their feedback. 7.2.2 Automatically providing kudos and critique rationales helped participants formulate their intuitive feedback Sometimes the generated kudos and critique rationales did not capture participants’ particular feedback on the chatbot, and so they would then write their own. Their feedback was often under-specified, and ConstitutionMaker helped convert this feedback into a clear and specific principle. For example, P4 was writing principles for VacationBot using ConstitutionMaker. During the conversation, they had told VacationBot that they were planning a week long vacation to Japan, to which the bot immediately responded with a comprehensive 7-day itinerary. P4 then wrote in their own critique: “This response is bad because it does not take into account the user’s interests.” The resulting principle was, “When the user mentions a location, ask them questions about what they are interested in before providing an itinerary.” This principle was aligned with what P4 had in mind, and reflecting on her experience using ConstitutionMaker, she stated, Finally, even when the resulting principle was not exactly what they had in mind, participants appreciated the useful starting point it provided. Along these lines, P11 explained, Overall, ConstitutionMaker helped participants specify their feedback into clear principles. 7.2.3 Generating principles from feedback helped users write clear, specific principles. “When I would critique or kudos, it would give examples of principles that were putting it into words a little bit better than I could about like what exactly I was trying to narrow down to here.” “It was easier to say yes-and with Tool A [ConstitutionMaker]. Where it [the generated principle] wasn’t all the way there, but I think it’s 50% of the way there, and I can get it to where I want to go.” 7.3 Participants’ Workflows Introduced Challenges with Writing Principles Participants struggled to find the right level of granularity for their principles, and the two conditions led to different problems in this regard. Both workflows had participants switch roles from end-user, where participants experimented with different user journeys, to bot designer, where they evaluated the bot’s responses to write principles. The more conversation-forward interface of the baseline blurred the distinction between these two roles. P3 explained that without the multiple bot outputs and principle elicitation features, And by leaning further into this user role, participants wrote principles that were more conversational, but under-specified. For example, while writing principles for FoodBot with the baseline, P11 wrote the principle “Be cognizant of the user’s dietary preferences.” What P11 really had in mind was a principle that specified that the bot should ask the user for their preferences and allergies prior to generating a meal plan. These underspecified principles often did not impact the bot’s responses and would frustrate participants while they used the baseline. “you can simulate yourself as the user a lot better in this mode [the baseline].” Meanwhile, while using ConstitutionMaker, the opposite problem occurred, where users’ workflows led to principles that were over-specified. For example, while working on VacationBot, P7 asked the model to help him narrow down a few vacation options, and the model proceeded to ask them questions (without any principle written specifying so). Appreciating that the model was gathering context, they selected a kudos that praised the model for asking about the user’s budget constraints prior to recommending a vacation destination. The resulting principle was, “Ask the user their budget before providing vacation options.” However, once this principle came into effect, the model’s behavior then anchored specifically to only asking for budget prior to making a recommendation. And so, this workflow of providing feedback at every conversational step, instead of for entire conversations, led to a principle that was too specific and impacted the bot’s performance negatively. While users generally appreciated ConstitutionMaker’s ability to form specific principles from their feedback, there were rare instances where the principles were too specific. Finally, in both conditions, by switching back and forth between the end-user to bot designer roles, participants would sometimes write principles that conflicted with each other. For example, while P2 was working on VacationBot with the baseline, they asked the bot for dog-friendly hotel recommendations in the Bay Area, and VacationBot responded with three recommendations. P2 wanted more recommendations and wrote a principle to “Provide >= 10 recommendations.” Later on in the conversation, P2 now had a list of dog-friendly restaurants, with their requisite costs, and he asked VacationBot which it recommends, to which it responded by listing positive attributes of all the hotel options. P2, who now wanted a decisive, single response wrote the following principle: “If I ask for a recommendation, give *1* recommendation only.” VacationBot, now with two conflicting principles on the number of recommendations to provide, alternated between the two. Ultimately, by providing feedback on individual conversational turns, participants ended up with conflicting principles. P8 imagined a different workflow, where he would experiment with full user journeys and then write principles: In summary, one’s workflow as they probe and test the model impacts the types of principles they produce and the challenges they face. “I think it might help me to actually go through the [whole] user flow and then analyze it as a piece instead of switching...it would allow me to inhabit one mindset [either bot designer or user] for a period of time and then switch mindsets.” 7.4 The Limits of Principles Some participants questioned if writing natural language principles was the optimal way to steer all aspects of a bot’s behavior. While writing a principle to shorten the length of the chatbot’s responses, P13 reflected, They imagined that aspects like the form of the model’s responses would be better customized with UI elements like sliders to adjust the length of the bot’s responses, or exemplifying the structure of the bot’s response (e.g., an indented, numbered list for recommendations) for the model to follow, instead of describing these requests in natural language. In a similar vein, P14 noticed that her principles pertained to different parts of the conversation, and as a list, they seemed hard to relate to each other. She wanted to structure and provide feedback on higher-level “conversational arcs,” visually illustrating the flow and “forks in the road” of the conversation (e.g., “If the user does X, do Y. Otherwise, do Z”). Principles are computational in a sense, and they dictate the ways the conversation can unfold; there might be better ways to let users author this flow, other than with individual principles. “It feels a little weird to use natural language to generate the principles...it doesn’t feel efficient, and I’m not sure how it’s going to interpret it.” This paper is under CC 4.0 license. available on arxiv

Efficiency and Intuition: Key Factors in User Preference for ConstitutionMaker

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 6 USER STUDY To understand (1) if the principle elicitation features help users write principles to steer LLM outputs and (2) what other kinds of feedback they wanted to give, we conducted a 14-participant within-subjects user study, comparing ConstitutionMaker to an ablated version without the principle elicitation features. This ablated version still offered users the ability to rewind parts of the conversation, but participants could only see one chatbot output at a time and had to write their own principles. 6.1 Procedure The overall outline of this study is as follows: (1) Participants spent 40 minutes writing principles for two separate chatbots, one with ConstitutionMaker (20 minutes) and the other with the baseline version (20 minutes), while thinking aloud. Condition order was counterbalanced, with chatbot assignment per condition also balanced. (2) After writing principles for both chatbots, participants completed a post-study questionnaire, which compared the process of writing principles with each tool. (4) Finally, in a semi-structured interview, participants described the positives and negatives of each tool and their workflow. The total time commitment of the study was 1 hour. The two chatbots participants wrote principles for were VacationBot, an assistant that helps users plan and explore different vacation options, and FoodBot, an assistant that helps you plan your meals and figure out what to eat. These two chatbots were chosen because they support tasks that most people are experienced with, so that participants could have opinions on their outputs and write principles. For both chatbots, participants were given the name and capabilities (Figure 1A), so as to focus predominantly on principle writing. Also, we balanced which chatbot went with each condition, so half of the participants used ConstitutionMaker to write principles for VacationBot, and the other half used the baseline for VacationBot. Finally, prior to using each version, participants watched a short video showing that respective tool’s features. To situate the task, participants were asked to pretend to be a chatbot designer and that they were writing principles to dictate the chatbot’s behavior so that it performs better for users. We wanted to observe their process for writing principles and see if the tools impacted how many principles they could write, so we encouraged participants to write at least 10 principles for each chatbot, to give them a concrete goal and to motivate them to write more principles. However, we emphasized that this was only to encourage them to write principles and that they should only write a principle if they thought it would be useful to future users. 6.2 Measurements and Analysis 6.2.1 Questionnaire. elicitation features help users write principles. Our questionnaire (Table 1) probes a few aspects of principle writing including participants’ perception of (1) how well the output principles effectively guide the bot, the diversity of the output principles, how easy it was to convert their feedback into principles, the efficiency of their principle writing process, and the requisite mental demand [9] for writing principles with each tool. To compare the two conditions, we conducted paired sample Wilcoxon tests with full Bonferroni correction, since the study was within-subjects and the questionnaire data was ordinal. We wanted to understand if and how well ConstitutionMaker’s principle To shed further light on which tool helped participants write principles more, we recorded the number of principles written for each condition. Moreover, to understand which of the principles elicitation features was most helpful, we recorded how often each was used during the experimental (full ConstitutionMaker) condition. To compare the average number of principles collected across the two conditions, we conducted a paired t-test. 6.2.2 Feature Usage Metrics. 6.3 Participants We recruited 14 industry professionals at a large technology company (average age = 32, 6 female and 8 male) via an email call for participation and word of mouth. These industry professionals included UX designers, software engineers, data scientists, and UX researchers. Eligible participants were those that had at least written a few LLM prompts in the past. The interviews were conducted remotely. Participants received a $25 gift card for their time. This paper is under CC 4.0 license. available on arxiv

Comparing ConstitutionMaker to Baseline: User Study Unveils Insights into Chatbot Principle Writing

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 4 CONSTITUTIONMAKER Inspired by our findings from the formative studies and workshop, we built ConstitutionMaker, an interactive web tool that supports users in converting their feedback into principles to steer a chatbot’s behavior. ConstitutionMaker enables users to define a chatbot, converse with it, and within the conversation, interactively provide feedback to steer the chatbot’s behavior. 4.1 Interface and Walk Through To illustrate how ConstitutionMaker works, let us consider a music enthusiast, Penelope, who would like to design a chatbot that helps you learn about and find new music, called MusicBot. She starts by entering the name of her bot and roughly describing its purpose in the “Capabilities” section of the interface (Figure 1A). She then starts a conversation with MusicBot, and after MusicBot’s introductory message, she asks to learn about punk music (Figure 1B). Fulfilling our first design goal, help users recognize ways to improve the bot’s responses, at each conversational turn, ConstitutionMaker provides three candidate responses from the bot (Figure 1D) that the user can compare and provide feedback on. Penelope peruses these candidate responses, and of the three responses, she likes the first one, as it invites the user to continue the conversation with a question at the end. She now wants to write a principle to help ensure that the chatbot will continue to do this in future conversations. Fulfilling D.2, help convert user feedback into principles, ConstitutionMaker provides three principle-elicitation features to support users in converting their feedback to principles: kudos, critique, and rewrite. Since Penelope likes the response, she selects kudos underneath it (Figure 1D), which reveals a menu with three automatically generated rationales on why the response is good, as well as a text field for Penelope to enter her own reason. After scanning the rationales, she selects the second, as it closely matched her own feedback, and subsequently a principle is automatically generated from that rationale (Figure 1C). The critique (Figure 1F) and rewrite (Figure 1G) principle elicitation features work similarly, where Penelope can select a negative rationale or rewrite the model’s response to generate a principle respectively. She then inspects the generated principle, decides that it captured her intention well, and decides not to edit it. Fulfilling D.3, enable easy testing of principle, she can then test to see if the chatbot is following her principle by rewinding the conversation (Figure 1H) to get a new set of candidate responses from the model. Ultimately, she decides to continue conversing with MusicBot, exploring different user journeys, and using the principle elicitation features to create a comprehensive set of principles. This paper is under CC 4.0 license. available on arxiv

ConstitutionMaker: Revolutionizing Chatbot Design Through User-Guided Principles

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 2 RELATED WORK 2.1 Designing Chatbot Behavior There are a few methods of creating and customizing chatbots. Earlier chatbots employed rule-based approaches to construct a dialogue flow [11, 29, 42], where the user’s input would be matched to a pre-canned response written by the chatbot designer. Later on, supervised machine learning approaches [43, 48] became popular, where chatbot designers constructed datasets consisting of ideal conversational flows. Both of these approaches, while fairly effective, require a significant amount of time and labor to implement, from either constructing an expansive rule set that determines the chatbot’s behavior or from building a large dataset consisting of ideal conversational flows. More recently, large language model prompting has shown promise for enabling easier chatbot design. Large, pre-trained models like Chat-GPT [28] can hold sophisticated conversations out of the box, and these models are already being used to create custom chatbots in a number of domains, including medicine [18]. There are a few ways of customizing an LLM-based chatbot, including prompt engineering and fine-tuning. Prompt engineering involves providing instructions or conversational examples in the prompt to steer the chatbot’s behavior [4]. To more robustly steer the model, users can also fine-tune [19] the LLM with a larger set of conversational examples. Recent work has shown that users would also like to steer LLMs by interactively critiquing its outputs; during the conversation they refine the model’s outputs by providing follow-up instructions and feedback [5]. In this work, we explore how to support users with this type of model steering: naturally customizing the LLM’s behavior through feedback, as they interact with it. A new approach to steering LLM-based chatbots (and LLMs in general), called Constitutional AI [1] involves writing natural language principles to direct the model. These principles are essentially rules, such as: “Do not create harmful, sexist, or racist content”. Given a set of principles, the Constitutional AI approach involves rewriting LLM responses that violate these principles, and then using these tuples of original and rewritten responses to fine tune the LLM. Writing principles could be a viable and intuitive way for users to steer LLM-based chatbot behavior, with the added benefit of being able to use these principles later to fine tune the model. However, relatively little is known about the kinds of principles users want to write, and how we might support users in converting their natural feedback on the model’s outputs into principles. In this work, we evaluate three principle elicitation features that help users convert their feedback into principles to steer chatbot behavior. 2.2 Helping Users Design LLM Prompts While LLM prompting has democratized and dramatically sped up AI prototyping [12], it is still a difficult and ambiguous process for users [31, 45, 46]; they have challenges with finding the right phrasing for a prompt, choosing good demonstrative examples, experimenting with different parameters, and evaluating how well their prompt is performing. [12]. Accordingly, a number of tools have been developed to support prompt writing along these lines. To help users find a better phrasing for their prompt, automatic approaches have been developed that search the LLM’s training data for a more effective phrasing [27, 32]. In the text-to-image domain, researchers have employed LLMs to generate better prompt phrasings or keywords for generative image models [3, 21, 22, 37]. Next, to support users in sourcing good examples for their prompt, ScatterShot [39] suggests underrepresented data to include in the prompt from a dataset, and enables users to iteratively evaluate their prompt with these examples. Similar systems help users source diverse and representative examples via techniques like clustering [6] or graph-based search [34]. To support easy exploration of prompt parameters, Cells, Generators, and Lenses [16] enables users to flexibly test different inputs with instantiations of models with different parameters. In addition to improving the performance of a single run prompt, recent work has also investigated the benefits of chaining multiple prompts together, to improve performance on more complicated tasks [40, 41]. Finally, tools like PromptIDE [33], PromptAid [23], and LinguisticLens [30] support users in evaluating their prompts, by either visualizing the data it produces, or its performance in comparison to other prompt variations. This work explores a novel, more natural way of customizing a prompt’s behavior through interactive critique. ConstitutionMaker enables users to provide natural language feedback on a prompt’s outputs, and this feedback is converted into principles that are then incorporated back into the prompt. We illustrate the value of helping users update a prompt via their feedback, and we introduce three novel mechanisms for converting users’ natural feedback into principles for the prompt. 2.3 Interactive Model Refinement via Feedback Finally, ConstitutionMaker is broadly related to systems that enable users to customize their outputs via limited or underspecified feedback. For example, programming-by-example tools enable users to provide input-output examples, for which the system generates a function that fits them [7, 35, 47]. Input-output examples are inherently ambiguous, potentially mapping to multiple functions, and these systems employ a number of methods to specify and clarify the function with the user. In a similar process, ConstitutionMaker takes ambiguous natural language feedback on the model’s output and generates a more specific principle for the user to inspect and edit. Next, recommender systems [2, 17, 25] also enable users to provide limited feedback to steer model outputs. One such system [17] projects movie recommendations on a 2D plane, which users can interactively raise or lower portions of it to affect a list of recommendations; in response to these changes, the system provides representative movies for each raised portion to demonstrate how it has interpreted the user’s feedback. Overall, in contrast to these systems, ConstitutionMaker leverages LLMs to enable users to provide natural language feedback and critique the model in the same way we would provide feedback to another person. This paper is under CC 4.0 license. available on arxiv

Chatbot Design: A Journey from Rule-Based Systems to Interactive Critique

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References ABSTRACT Large language model (LLM) prompting is a promising new approach for users to create and customize their own chatbots. However, current methods for steering a chatbot’s outputs, such as prompt engineering and fine-tuning, do not support users in converting their natural feedback on the model’s outputs to changes in the prompt or model. In this work, we explore how to enable users to interactively refine model outputs through their feedback, by helping them convert their feedback into a set of principles (i.e. a constitution) that dictate the model’s behavior. From a formative study, we (1) found that users needed support converting their feedback into principles for the chatbot and (2) classified the different principle types desired by users. Inspired by these findings, we developed ConstitutionMaker, an interactive tool for converting user feedback into principles, to steer LLM-based chatbots. With ConstitutionMaker, users can provide either positive or negative feedback in natural language, select auto-generated feedback, or rewrite the chatbot’s response; each mode of feedback automatically generates a principle that is inserted into the chatbot’s prompt. In a user study with 14 participants, we compare ConstitutionMaker to an ablated version, where users write their own principles. With ConstitutionMaker, participants felt that their principles could better guide the chatbot, that they could more easily convert their feedback into principles, and that they could write principles more efficiently, with less mental demand. ConstitutionMaker helped users identify ways to improve the chatbot, formulate their intuitive responses to the model into feedback, and convert this feedback into specific and clear principles. Together, these findings inform future tools that support the interactive critiquing of LLM outputs. CCS CONCEPTS → • Human-centered computing → Empirical studies in HCI; Interactive systems and tools; • Computing methodologies Machine learning. KEYWORDS Large Language Models, Conversational AI, Interactive Critique 1 INTRODUCTION Large language models (LLMs) can be applied to a wide range of problems, ranging from creative writing assistance [8, 26, 36, 44] to code synthesis [13, 14, 20]. Users currently customize these models to specific tasks through strategies such as prompt engineering [4], parameter-efficient tuning [19], and fine-tuning [10]. In addition to these common methods for customizing LLMs, recent work has shown that users would also like to directly steer these models with natural language feedback (Figure 2A). More specifically, some users want to be able to critique the model’s outputs to specify how they should be different [5]. We call this customization strategy interactive critique. When interacting with a chatbot like ChatGPT[1] [28] or Bard[2], interactive critique will often alter the chatbot’s subsequent responses to conform to the critique. However, these changes are not persistent: users must repeat these instructions during each new interaction with the model. Users must also be aware that they can actually alter the model’s behavior in this way, and must formulate their critique in a way that is likely to lead to changes in the model’s future responses. Given the potential value of this mode of customizing there is an opportunity to provide first-class support for empowering users to customize LLMs via natural language critique. In the context of model customization, Constitutional AI [1] offers a specific customization strategy involving natural language principles. A principle can be thought of as a rule that the language model should follow, such as, “Do not create harmful, sexist, or racist content”. Given a set of principles, a Constitutional AI system will 1) rewrite model responses that violate principles and 2) fine-tune the model with the rewritten responses. Returning to the notion of interactive critique, one can imagine deriving new or refined Constitutional AI principles from users’ critiques. These derived principles could then be used to alter an LLM’s prompt (Figure 2B) or to generate new training data, as in the original Constitutional AI work. While this recent work has shown principles can be an explainable and effective strategy to customize an LLM, little is known about the human processes of writing these principles from our feedback. From a formative study, we discovered that there are many cognitive challenges involved in converting critiques into principles. To address these challenges, we present ConstitutionMaker, an interactive critique system that transforms users’ model critiques into principles that refine the model’s behavior. ConstitutionMaker generates three candidate responses at each conversational turn. In addition to these three candidate responses, ConstitutionMaker provides three principle-elicitation features: 1) kudos, where users can provide positive feedback for a response, 2) critique, where users can provide negative feedback for a response, and 3) rewrite, where users can rewrite a given response. From this feedback, ConstitutionMaker infers a principle, which is incorporated in the chatbot’s prompt. To evaluate how well ConstitutionMaker helps users write principles, we conducted a within-subjects user study with 14 industry professionals familiar with prompting. Participants used ConstitutionMaker and an ablated version that lacked the multiple candidate responses and the principle-elicitation features. In both cases, their goal was to write principles to customize two chatbots. From the study, we found that the two different versions yielded very different workflows. With the ablated version, participants only wrote principles when the bot deviated quite a bit from their expectations, resulting in significantly fewer principles being written, in total. In contrast, in the ConstitutionMaker condition, participants engaged in a workflow where they scanned the multiple candidate responses and gave kudos to their favorite response, leading to more principles overall. These different workflows also yielded condition-specific challenges in writing principles. With the ablated version, users would often under-specify principles; whereas, with ConstitutionMaker, users sometimes overspecified their principles, though this occurred less often. Finally, both conditions would sometimes lead to an issue where two or more of the principles were in conflict with one another. Overall, with ConstitutionMaker, participants felt that their principles could better guide the chatbot, that they could more easily convert their feedback into principles, and that they could write principles more efficiently, with less mental demand. ConstitutionMaker also supported their thought processes as they wrote principles by helping participants 1) recognize ways responses could be better through the multiple candidate responses, 2) convert their intuition on why they liked or disliked a response into verbal feedback, and 3) phrase this feedback as a specific principle. Collectively, this work makes the following contributions: • A classification of the kinds of principles participants want to write to steer chatbot behavior. • The design of ConstitutionMaker, an interactive tool for converting user feedback into principles to steer chatbot behavior. ConstitutionMaker introduces three novel principleelicitation features: kudos, critique, and rewrite, which each generate a principle that is inserted into the chatbot’s prompt. • Findings from a 14-participant user study, where participants felt that ConstitutionMaker enabled them to 1) write principles that better guide the chatbot, 2) convert their feedback into principles more easily, and 3) write principles more efficiently, with less mental demand. • We describe how ConstitutionMaker supported participants’ thought processes, including helping them identify ways to improve responses, convert their intuition into natural language feedback, and phrase their feedback as specific principles. We also describe how the different workflows enabled by the two systems led to different challenges in writing principles and the limits of principles. Together, these findings inform future tools for interactively refining LLM outputs via interactive critique. [1] https://chat.openai.com/ [2] https://bard.google.com This paper is under CC 4.0 license. available on arxiv

Interactively Critiquing Large Language Models by Converting Feedback into Principles 

মতামতকে নীতিতে রূপান্তর করে ইন্টারেক্টিভভাবে বড় ভাষার মডেলের সমালোচনা করা

Interaktive Kritik großer Sprachmodelle durch Umwandlung von Feedback in Prinzipien

Criticar interactivamente grandes modelos de lenguaje convirtiendo la retroalimentación en principios

Critique interactive de grands modèles de langage en convertissant les commentaires en principes

फीडबैक को सिद्धांतों में परिवर्तित करके बड़े भाषा मॉडलों की अंतःक्रियात्मक रूप से आलोचना करना

フィードバックを原則に変換することで大規模な言語モデルを対話的に批判する

피드백을 원칙으로 변환하여 대규모 언어 모델을 대화형으로 비판

Crítica interativa de grandes modelos de linguagem, convertendo feedback em princípios

Интерактивная критика больших языковых моделей путем преобразования обратной связи в принципы

Geri Bildirimi İlkelere Dönüştürerek Büyük Dil Modellerini Etkileşimli Olarak Eleştirmek

Phê bình một cách tương tác các mô hình ngôn ngữ lớn bằng cách chuyển đổi phản hồi thành nguyên tắc

通过将反馈转化为原则来交互式批评大型语言模型

Let's learn about via these 57 free stories. They are ordered by most time reading created on HackerNoon. Visit the to find the most read stories about any technology. Ico /Learn Repo Talk-in' 'bout, hey now hey now I-ko, I-ko, un-day
Jock-a-mo fee-no ai na-né, jock-a-mo fee na-né
Look at my king all dressed in red I-ko, I-ko, un-day
I bet-cha five dol-lars he'll kill you dead, jock-a-mo fee na-né Also, Initial Coin Offerings. 1. 3 Things Not to Do When Bitcoin is Going Down Photo by Ian Stauffer on Unsplash 2. Understanding IBCO: Can Another Crypto-Funding Mechanism Yield Accountable Results? TLDR: IBCO is the new primitive for crypto projects funding, solving for the main issues of ICOs such as lack of accountability, lack of liquidity and flawed price discovery mechanics for the token itself. 3. My Journey in CryptoCollectibles (so far) A journey marked by crypto collectibles and highlights of lessons learned along the way. 4. Are NFTs the Next ICO Fraud? Want to buy virtual artwork for $1.98M? 5. Why Metaverses are the Next Big Trend After NFTs The Metaverse is a rocket ship that will propel the popularity and application of blockchain technologies and cryptocurrencies to a whole new level. 6. Top 75+ ICO Marketing Agencies There are many stages to launching an ICO and the needs of the business vary depending at what stage they’re on. For each of these stages we’ve highlighted what services are needed and organized our Top 75 list according to their offered services across five main categories: 7. Raising Funds for Blockchain Projects is a Long and Winding Road Raising funding for blockchain projects presents many challenges. This article lists some of them. 8. 10-Step Checklist for Assessing an Initial Coin Offering (ICO) Project Dye to the growing demand for educating new crypto investors, we’ve put together the following, 10 definitive factors for assessing a successful ICO project. 9. From Utility to Security: The U2S Model for Converting Shitcoins into Securities During the ICO era, a countless number of projects managed to rapidly raise capital in a short period of time. You probably know about those that managed to raise capital and just disappeared, never building a product promised to investors while their utility tokens became a burden for investors. It’s hard to admit but there is nothing you can do to change the historical outcome. 10. Top 3 Things You Need to Know About Initial Game Offerings (IGOs) IGO become one of the latest crypto trends in the crypto space with the rise of blockchain games or GameFi, and here are top 3 things you need to know about it. [11. 2017 Vs 2020 in crypto: any chance to replicate "success"?](https://hackernoon.com/2017-vs-2020-in-crypto-any-chance-to-replicate-success-ocm3zdo) It is not a big secret, that a lot of market players are trying to compare the current situation on the crypto market with the phenomenon we saw in 2017. Back then, in October, the BTC price was 2 times lower and ICO made the market super attractive for scammers (so as  DeFi now). Should we really be prepared for a rally starting from now? 12. Top 5 Methods of Running an Effective Cryptocurrency Exchange Anti Danilevski, KickEX CEO and founder, talks about Top 5 Methods of Running an Effective Cryptocurrency Exchange based on his own experiences in the field. 13. Number of STOs May Double While IEOs Will Almost Triple By Q4 2019 With regulators clamping down uncontrolled blockchain funding, I expect the number of ICOs to drop 90% from 980 registered in Q1 2018 to around 100–110 in Q4 2020. At the same time, regulatory compliance would allow STOs and IEOs to flourish. 14. My Blockchain Adventures in Africa In December 2017 I had an opportunity to discover the Kenyan based Nurucoin ICO and write up a spotlight article on it. As sometimes happens, I develop an affinity for the person/company that I talked to when putting these articles together and in this case, I kept in contact with CEO Isaac Muthui. I would answer questions for him and point out information to him that would be helpful for his project. After a couple months he invited me to come and visit their company in Kenya, they were interested in my opinions on technology they were building. I agreed to come because, why not, a trip to Africa. I wasn’t compensated for the trip or for this article, but my expenses were covered, I mention that in the interest of full disclosure. What I discovered in Kenya and about Isaac and the goals for Nurucoin, got me thinking in some extremely big pictures. 15. "Experience Imposes the Strongest Psychological Inertia" - Andrey Sergeenkov, Noonies Nominee Andrey Sergeenkov from Russia has been nominated for a 2020 Noonie in the Future Heroes, Technology, and Decentralization categories. Andrey has been nominated for four separate awards and is anxiously awaiting the day when DeFi becomes mainstream. 16. Mr. Market Meets Mr. ICO Ben Graham was a towering intellect. He wrote a book which has inspired legions of hedge fund managers and smaller investors to wealth, and quite interestingly, they all took the basic concepts laid out in this book, but applied them in radically different ways. Warren Buffett is Ben Graham’s best known pupil, but there are legions of others, the so-called “Super Investors of Graham and Doddsville”. 17. Get Used to Building a .Net core MVC Application on Mac Visual Studio Creating .Net MVC project using Visual Studio Mac 18. Saving Cryptocurrency from Scammers: Interview with Crypto Vigilante I think the ICO run is past us. We will not see a crazy market like we did in 2017. Scammers are very sophisticated and are always coming up with new ways to scam. 19. Can IDOs #RightTheWrongs of ICOs and IEOs? Crypto startups are stacking up to launch their IDOs on DEX platforms like Polkastarter, DuckSTARTER, SpacePort, BSCPad and a few other leading platforms. 20. A Safer Way of Investing in IDOs with Priority Projects Priority Projects - Kommunitas' way of making IGOs less risky and promoting high quality projects. 21. Life After ICO: 4 New Ways to Launch a Token The history and end of the ICO hype are well known: many projects turned out to be unfeasible or fraudulent, and many people lost their money. 22. What Crypto Should You Invest In? When you work in the blockchain space, you'll hear one question again and again, whether it's from your barber, your friends, or a social media DM: 23. Founder Interviews: Mohammad Mazen of Burency It is no longer a secret that the United Arab Emirates are technology pioneers, and their action plan for the development of Blockchain technology within companies and government has become a clear priority. 24. Blockchains for Crowdfunding It is no secret that Crowdfunding hold a tremendous potential in helping entrepreneurs and creators in pushing their projects forwards and gathering much needed financial resources. In 2018, The crowdfunding market size was valued at 10.2 billions dollars and was set to almost triple by 2025. 25. Everything Changes and Nothing Does… In the past few months, I have met and spoken to many people from the blockchain space. Enthusiasts, professionals, enthusiasts who pretended to be experts, people who were predicting the future of the decentralization and those who were super excited to hear about it for the first time… Moon and Lambo guys, crypto traders, blockchain startup founders…. people from the traditional finance who were telling me to run away before it’s too late because all this is just a bubble… 26. All About Initial DEX Offerings and Why They Matter A new method of investing in crypto startups is on the rise. Learn what an IDO is and how DeFi users and new projects are leveraging IDOs to reach new heights. 27. As ICOs Wane, Big Money Squeezes Out The Little Guy In 2017, the Initial Coin Offering (ICO) exploded onto the global scene and upended the traditional method of funding an early-stage business. For a while, many believed that we were witnessing the dawn of a more egalitarian financial era; one where anyone with a good idea and the ability to make it a reality could access easy money and a shot at success. As it turned out, it was too good to be true. 28. Looking Back At The Companies That Led To The ICO Boom One popular narrative about 2017's ICO mania says it was fueled by naive investors duped into throwing money at projects that were either incapable or had no intention, of delivering. 29. BTC, ETH, ICOs, and now - DeFi: How far Have We Come? Bitcoin caught the attention of technologists and anarchists alike following the 2008 Global recession. Ethereum took Bitcoin’s ‘Digital Gold’ status and gave us smart contracts. ICOs took smart contracts and gave us supply chain management solutions for problems that we never knew existed. 30. The Magic of Altcoins: How do I Know if a Coin is Worthy of Attention? There are more and more altcoins. But how do you know it's not a scam? How do you know if a coin is worth paying attention to? We tell you in this column. 31. How Does Pandemic Impact Company Valuation With the pandemic induced havoc, how is that the stock market is not down more? And what effect will the economic downturn have on the valuation of startups? 32. The IEO Gamble: Does it Raise Capital or only Trouble? Graphics by Regex SEO 33. Applying Complexity Theory to Valuation The last of three articles on corporate valuations and capital structures. 34. What Do Regulators and Investors Want To See In Your White Papers? In September 2017, the Office of Financial Conduct Authority of the UK (FCA) issued a warning about the risks associated with the ICO. In particular, they directly pointed out to the weak level of White Papers, and did describe it like the risk of the “imperfect documentation”. 35. Interview with @MarvinSteinberg On Asset Tokenization, Startups, Investors and More The concept of asset tokenization is still in its formative state, as companies are just coming to terms with the idea of capitalizing on blockchains to distribute shares and assets to interested investors. While this is a given, the concept is looking ever more likely to hit the ground running in 2020. This eventuality is bound to contribute to the potency of STOs as a fundraising mechanism for startups and established companies. 36. The Ugly Truth Behind IEOs and What to Expect in 2020 Let’s face it, you won’t be able to conduct a successful IEO unless you have a fully-operational and fully compliant prototype and unless you’ve clinched a listing deal with a major exchange with actual liquidity within its community of retail investors, and traders. 37. MO’s Algorithm: Efficient Way to Solve Offline Range Query Problems MO’s Algorithm aka Square Root Decomposition, a very efficient and easy technique to solve Range Query Problems (RQP). For MO’s Algorithm to work, the RQP has to be offline. In this post, we will understand about RQP, Offline RPQ, Naive Approach to solve RQP and an Efficient Approach using MO’s Algorithm. What is Range Query Problem? 38. Looking Back At ICOs At The Time Of The DeFi Hype Train Although I have been in the blockchain space for more than 3 years, I am not from tech, which is probably why this  post will be looking more into the story and the human aspects of ICOs. 39. IEO (Initial Exchange Offering): What Kind of Beast Is It? In 2016, ICO became the main way of financing innovation and blockchain startups. The first project, released on ICO, was Mastercoin in 2013. After it, thousands of projects attracted funding through the emission of tokens. According to ICODATA in 2017, the volume of investments raised through ICO (Initial Coins Offerings) were more than $6.2 billion, in 2018 — about $8 billion. After analyzing the data from the TokenData, it can be concluded that in 2018, unfortunately majority of the projects turned out to be a scam. 40. GAME of CRYPTOS  [Chapter 5 ] - The Token Apocalypse It is the year of the Lord of 2017. 41. How To Identify Scam ICOs Scammers are nothing new. They have been around us for ages utilizing different mediums and resources to set traps. Progressing civilization and advancement in technologies create different loopholes that later get exploited and we, the people, pay the price. Crypto is one such universe, full of scams that are hard to spot. 42. Did Your Shitcoin Break the Law? [Part 2] In the movie Alladin, Jafar wishes to be a genie because of their incredible might and power. Alladin proclaims, “You wanna be a genie? You got it!” 43. Working Towards a Sustainable Ecosystem Beyond Tokenization Many industries are on the brink of the next technological revolution in record keeping. Ten years after Bitcoin made its splash, we’re seeing many inspired by some of the benefits promised by the technology outside of the money use case: 44. IEOs are a Crypto Has-Been While some endorse IEOs, Kick Ecosystem’s CEO Danilevski argues that they’re a dangerous scam enterprise which will soon be replaced by the much more secure STOs and honest utility token sales. 45. Crypto Staking: Assessing the Risks and Profitability of USD Neutrino, Cosmos and others Crypto industry keeps generating new investment tools – only to be discarded  in disappointment a year or two later. The latest trend is staking - delegating coins to to earn validation rewards on the blockchain. Is it a legitimate passive income scheme? And what should you choose - regular PoS coins like Cosmos or stakable stablecoins? 46.  "IEO Will Continue to be Mainstream Even With IDO And DeFi on The Rise" - Kiana Shek The current push for DeFi has not only changed the way we engage with our everyday financial needs but has also introduced new means of accessing, supporting, and investing in new projects. Tagged as Initial DEX Offerings (IDO), the possibility of bypassing restrictions when identifying and investing in projects has improved crypto funding. However, this does not mean that other models of crowdfunding have become redundant. For what is worth, Initial Exchange Offering (IEO) is still the go-to option for projects that prefer to capitalize on established launchpads and introduce a more diverse array of crypto products not necessarily restricted to the financial sector. 47. Is Binance Launchpad a Breeding Ground for Pump and Dumps? Back in 2017, Initial Coin Offerings (ICOs) were all the rave. Much like Initial Public Offerings, (IPO), ICOs act as a way to raise funds by selling a quantity of cryptocurrency in the form of coins to speculators and investors. 48. What Is a (Crypto)Coin Burn? Bitcoin is the main cryptocurrency many people know, understand, and are familiar with. It is also one of the few digital currencies that has no individual or group that is in charge of the blockchain project. What has come about since Bitcoin was created is a cryptocurrency ecosystem where there are a number of companies in charge of blockchain token projects. 49. The ICO to STO Evolution is Complete. So, What’s the Difference? In the blockchain industry, there is now a lot of talk about replacing the ICO with a new, more legitimate method of attracting investment, called an STO. However, this definitely is not something completely new. During the heyday of the ICO, entering the market with security tokens was not very popular, and even now, many projects, and the investors themselves, do not always understand the difference. In this article, I will explain what makes an STO and an ICO fundamentally different. 50. Making Sense of the Difference Between IPO, ICO, IEO, and STO? The advent of blockchain technology gave life not only to a new profitable financial asset, but also a new business model of funding startups. An increasing number of new companies are trying to use cryptocurrency to finance their projects. Unlike an IPO, which is a traditional and widely accepted form of external financing, STO, IEO, and ICO allow companies to start fundraising at a much earlier stage of development and the latter two avoid strict regulatory control. Already today, as a result of ICO, more than 1000 cryptocurrencies appeared on the market, and their number continues to grow. The practice of ICO has become so widespread that, in the opinion of many experts, it began to resemble the events of the dotcom boom of the late 90s and could lead to the same destructive consequences. 51. What is tokenomics and how it can make or break your ICO Imagine you are the ruler of a small country. All residents of your country use the barter system to trade goods and services. Now you find barter to be a cumbersome process because of the following reasons: 52. WTF is Exchange Staking? Introduction 53. How to Find the Best IDOs (Part 1) How to find the best IDOs 54. They Told us ICOs are Dead. We Beg to Differ. ICO or Initial Coin Offering is a type of fundraising done with cryptocurrencies. Modeled after the traditional IPO (Initial Public Offering), ICOs are ways for startup cryptocurrency companies to sell their underlying token in exchange for Bitcoin, Ether or Fiat. 55. Can the Combination of STO + IEO Become the New Step in Crowdfunding Evolution? Content of the Article: 56. Unraveling The Budget Of A Crypto Startup: Behind The Secrecy In project management, there are several key success criteria for the possible implementation of any project: from building a plant, developing a software product, opening a new supermarket to something else. They are time, budget and quality. 57. CRYPTO REVIEW- SUBSTRATUM (SUB) Fundamentals Check: Thank you for checking out the 57 most read stories about Ico on HackerNoon. to find the most read stories about any technology. Visit the /Learn Repo

57 Stories To Learn About Ico

In the rapidly evolving realm of Web3, where innovation meets practicality, I had the pleasure of sitting down with Kevin Lu, the dynamic Co-Founder of Scalene and its flagship product, Friendzone (FZE). With a background rich in blockchain development and strategic growth, Kevin has spearheaded initiatives that have reshaped the landscape of and Social-Fi. Today, we delve into the intricate layers of Friendzone, exploring the vision, mechanics, and future prospects of this groundbreaking platform. Web3 Revolutionizing Social-Fi: A Deep Dive with Kevin Lu, Co-founder of Scalene & Pioneer Behind Friendz 1. Ishan Pandey: Kevin, it's an absolute honor to have you with us. To kick things off, can you share with our Hackernoon readers the story behind Scalene and its transition to a Web3 Venture Builder? Ishan, thank you for this opportunity to share more about Scalene and Friendzone. Our vision to onboard the next wave of Web3 builders and consumers was co-founded with a self-funded investment of $750K USD. Kevin Lu: Our journey began by fostering the talents of Web3 enthusiasts; we launched two core products, an Academy Platform and Talent Engine, that offered placement programs for top-tier talent into exceptional teams or a showcase opportunity at an industry-leading event. To debut our products, we co-hosted Glitch Hackathon, the first multi-chain developer festival in South Korea, where we supported over 300 builders and 1200 conference attendees. This initiative alongside Trinity Venture Studios, has evolved into a joint venture, reinforcing our commitment to developing emerging talent but also paved the way for our strategic growth. We also had the honor of contributing to GM VN, the largest Web3 Conference in Vietnam, as the exclusive hackathon and builder partner. Scalene alumni have joined some of the fastest-growing Web3 teams in 2023, including Infinex and Entangle Protocol. Our shift at Scalene to consumer-focused and mass-market products is a strategic answer to the changing landscape of Web3. Our goal is to draw a new wave of users and transform the digital world with impactful, user-centric experiences. Our debut and flagship product, Friendzone (FZE), marks a significant milestone in Scalene's journey. It embodies and is the first stepping stone on our mission to empower the entire consumer frontier of Web3, capitalizing on the now matured infrastructure landscape, leveraging advancements made available recently like ERC4337 and Privy. With Friendzone, we're championing a whole new ecosystem of social applications similar to DeFi Legos, which we refer to as SoFi Legos. In this ecosystem, every interaction transcends mere connectivity; it becomes a source for value creation, unlocking possibilities previously unimagined through composability, open-source, and decentralized collaboration. This evolution towards consumer-friendly and widely appealing products reflects our strategic foresight in adapting to the evolving dynamics of Web3, intending to attract an entirely new demographic of users and reshape the digital realm with engaging user-driven experiences. 2. Ishan Pandey: Friendzone is poised to be a notable player in the Social-Fi space. Based on the information available about its planned launch, could you shed light on the expected influence of Friendzone’s adaptive bonding curve and staking mechanism in this domain? How do you envision these features distinguishing Friendzone in the Social-Fi landscape? Drawing on our teams' experience in major DeFi, infrastructure, and middleware - we're proud to pioneer unique approaches within the Social-Fi domain. Our innovation of an adaptive bonding curve in our protocol design sets us apart as the first Web3 team to integrate a responsive pricing mechanism based on supply, demand, and reputation. Tying this with one of the core values behind Web3, ownership, we are excited to also be the first Web3 social project to implement native reward distribution through staking. Kevin Lu: Our core Unique Selling Points (USPs) revolve around these innovations. The essence of building a truly social Web3 product lies in embracing mechanisms that are inclusive and engaging for all users, regardless of their financial status or the timing of their platform engagement. We believe that limiting mechanisms, which could be perceived as 'anti-social,' have no place in the creation of a genuinely social Web3 environment - incentives are misaligned with speculation. The success of a Web3 project isn't solely dependent on intuitive UI/UX or an extensive feature set. The foundation of a project's longevity and impact lies in its protocol design and incentive loops. If these aspects are rudimentary or flawed, the project risks fizzling out after a speculative frenzy or worse, burning early users of the platform. At the heart of an exceptional Web3 project is a deep understanding of protocol, incentive design, and the unique financialization opportunities that crypto offers to leverage the resultant network effects. This understanding is the driving force behind building Friendzone from the ground up. We've observed numerous pitfalls in the current designs of , and we speak with users daily to understand their needs and challenges. This user-centric approach informs our development process, ensuring that we're not just another Social-Fi platform but paving the way for the next steps of evolution in this sector. SoFi platforms The features of Friendzone – particularly our adaptive bonding curve and staking mechanism – don't just differentiate us from other Social-Fi platforms; they set a new industry benchmark. These features are designed to foster a more inclusive, dynamic, and engaging Social-Fi environment. As we continue to iterate and refine Friendzone, our goal is to establish the industry standards for how Social-Fi projects should be conceptualized and executed, always with the protocol and incentive design at the forefront. 3. Ishan Pandey: You've had a remarkable journey with Band Protocol, marked by significant achievements. How have these experiences influenced your approach and vision for Scalene and Friendzone? My time at Band Protocol, complemented by my involvement in venture investments and advisory roles, was crucial in shaping my vision for Scalene and Friendzone. In Web3, the impact is often born from hands-on experience rather than just theory. The Web3 environment is ever-shifting, and practical experience is key. At Band Protocol, we reached significant milestones like securing $25B in total value and working with a wide spectrum of teams, from startups to mid-stage companies and enterprises like Google Cloud. These experiences have provided me with a deepened understanding of protocol and incentive design in Web3. Kevin Lu: Drawing from these rich experiences, coupled with the expansive network accrued over the last six years, provides Friendzone the competitive edge to create a Social-Fi platform that stands out and is here to stay in the rapidly evolving Web3 space. The key learning here is the importance of adaptive and user-centric product design, which is central to our vision. By integrating advanced incentive mechanisms and fostering a community-driven environment, we're addressing the gaps in current Social-Fi offerings. Our focus is not just on creating a platform but on cultivating an ecosystem where every interaction adds value, both socially and economically. 4. Ishan Pandey: The concept of monetizing social graphs is fascinating. Can you explain how Friendzone enables creators and users to transform their social engagement into tangible, monetizable assets? Our platform's core creator feature is the Custom Creator Controls, which provide end-to-end flexibility for content creators regardless of size. They can offer a variety of content, bounties, and experiences, setting their own terms and pricing for access directly within the app. This could range from Pay-Per-View (PPV) content to unique experiences or even exploring new concepts in social engagement. Our goal is to empower creators, brands, companies, and communities to tap into their audience base that is eager to support them more actively and intimately. Kevin Lu: During our version 1 incentivized beta launch, we're committed to engaging closely with our users. We believe in a hands-on approach to understanding their needs and preferences. This ongoing dialogue is crucial for refining our monetization streams. We're focusing on solving critical issues faced by creators and users in the monetization space and are excited to delve into creative markets that have yet to be defined. As we identify these highly demanded problems and feature requests, we plan to continuously roll out and expand our custom creator access controls. This iterative process ensures that we are not only responding to current market needs but are also ahead of the curve in anticipating and creating new opportunities. Our vision with Friendzone is to make social engagement not just interactive but also rewarding, transforming the way value is created and shared in the social graph. 5. Ishan Pandey: In terms of technology and user experience, what sets Friendzone apart from other Social-Fi platforms like Friendtech and Stars Arena? What fundamentally differentiates Friendzone from other Social-Fi platforms is our commitment to pioneering the capabilities of Web3 and industry growth. We consciously chose not to take the shortcut of forking existing projects. Unlike many SoFi projects, there isn’t a single line of code or element of protocol design in Friendzone that has been borrowed from existing models. This distinction is critical in the context of Web3 Social-Fi, which is still extremely early; forking often leads to mere value extraction from followers rather than creating meaningful engagement and value for both creators and users. Kevin Lu: Our approach is not just about avoiding reputational risks for creators, who, in an ideal scenario, should avoid having multiple profiles across various SoFi platforms of the same nature. It’s about aligning incentives with the true spirit of Web3 — moving away from mere speculation and towards creating genuine, sustainable value that is equitable and accessible. This ethos is why we are preparing for cross-chain capabilities by 2024, already having connections with various blockchain networks to enhance our platform’s reach and utility. At Friendzone, every aspect – from protocol level and incentive design to our product feature set, native reward distribution through staking, and, for the first time in Web3, an adaptive bonding curve – is a result of grass-roots efforts from our team. Our focus isn’t on capitalizing on short-term speculative frenzies. Instead, we’re strategically playing the long game, aiming for broader user distribution and establishing Friendzone as a truly social platform. Our commitment is to build a platform that not only stands the test of time but also reshapes the way social interactions and value creation are perceived in the digital realm. Ishan Pandey: Looking at the future, what exciting developments can we anticipate from Friendzone, particularly in terms of features like 'Anti-Chips' and the decentralized reputation system? The future of Friendzone is positioned to bring a fresh take in the realm of on-chain reputation, a challenge that still remains mostly unsolved in the blockchain space. Our perspective is that solving on-chain reputation requires a multifaceted approach, integrating factors like on-chain transactions, credibility, and social graphs. Friendzone's contribution in this space is our focus on leveraging social standing and social graphs as a building block to a comprehensive on-chain reputation system. Kevin Lu: An interesting concept we’re researching is ‘Anti-Chips,’ a feature designed to foster what we call 'fair market reputation.' This concept is based on allowing users to interact with reputations in a manner similar to market dynamics – users can 'bet' for (long) or against (short) someone's reputation. This is based on publicly available information, social graphs, and digital interactions, offering a transparent and open-market perspective on reputation. By integrating these considerations, we aim to create a decentralized reputation system that is both robust and reflective of real-world social dynamics - one that is permissionless, and resistant to Sybil attacks and exploits. We're excited about the potential of Anti-Chips to rethink the way reputation is built and perceived in the social world. Our conversations with CL207 have been instrumental in refining these early ideas, and we look forward to rolling out this feature, as it represents a significant step forward in our mission to enhance the utility and experience of social interactions on Friendzone. Ishan Pandey: Scalene's collaboration with giants like Google Cloud and Immutable is impressive. How do these partnerships enhance the capabilities and reach of your projects? : Our collaboration with key team members from Google Cloud and Immutable has been a significant milestone for Scalene. These relationships are a testament to our credibility and the strength of our team as Web3 native builders. Both Andy, our CTO and Co-Founder, and I have been immersed in the Web3 space for nearly six years, and our conviction in this industry is unwavering. We've found our passion and purpose in Web3, and these relationships reinforce our dedication and the trust we've earned in the industry. Kevin Lu We are incredibly humbled and grateful for the support of all our partners and advisors, especially during the formative stages of Scalene when we were launching our Academy Programs. Their backing played a crucial role in our early successes and in building our continued reputation in the Web3 community. Moreover, these relationships have opened significant doors for us, which is particularly powerful as we develop Friendzone. Social interaction is a fundamental aspect of human nature, and having the support of industry leaders boosts our ability to not only establish Friendzone as a leading consumer platform but also to extend its reach to support brands, companies, enterprises, and DAOs. Ishan Pandey: Finally, for our aspiring Web3 builders and enthusiasts, what advice would you offer, drawing from your extensive experience in this rapidly evolving field? Dive deep into the Web3 industry, not just as an observer, but as an active participant. Working in the field exposes you to its nuances and challenges, cultivating you into a Web3 native. It's about gaining hands-on experience and understanding the ecosystem from the inside out. Kevin Lu: Build something. Web3 is a space of endless possibilities, and hackathons are a great way to start. They often highlight gaps in the sector, offering opportunities to fill these with your ideas. Plus, there are prizes and recognition if you place well, which is merit to your skills and innovative thinking. Innovate: Engage on platforms like X/Twitter and Discord. Web3 is unique in that it offers visibility and direct access to a wide range of professionals, from founders to executives to thought leaders. It's a space where your voice can be heard and where you can learn directly from the experiences and insights of others. Interact: Last but not least, never ever hesitate to ask for help. Stay tuned for an enlightening journey through the digital landscape with Kevin Lu. Don’t forget to like and share the story! : Vested Interest Disclosure This author is an independent contributor publishing via our . Be it through direct compensation, media partnerships, or networking, the author has a vested interest in the company/ies mentioned in this story. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR brand-as-author program

Stories, News & Mentions about FORMOTIV

Formotiv’s Stories on HackerNoon