ForMotiv is the first-ever Digital Behavioral Intelligence (DBI) platform

ForMotiv

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 3 FORMATIVE STUDY To understand how to support users with writing principles for chatbots, we conducted (1) a one-hour formative study, where we observed eight industry professionals write principles for chatbots of their choice. These participants all had prompting experience. Two participants are designers and six are software engineers, all at a large technology company. During the workshop, participants used an early version of ConstitutionMaker, without principle elicitation features. They spent 25 minutes writing principles for their chatbot. Afterwards, we discussed the difficulties they faced while writing principles. Finally, we collected the principles they wrote and classified them to understand the kind of principles they wanted to write. 3.1 Design Goals In this section, we summarize a set of three design goals for ConstitutionMaker we established from the formative workshop and subsequent think-alouds. D.1 Today’s LLMs are quite sophisticated, and even with just a preamble describing how the bot should behave, the chatbot can hold a convincing conversation. Because of this, participants mentioned that it was sometimes hard to imagine how the chatbot’s responses could be improved. This did not mean, however, that they thought the chatbot’s response was perfect, but instead passable and without any glaring errors. Therefore, to help participants recognize better kinds of responses to steer the chatbot to, our first design goal was to provide multiple candidate responses from the chatbot at each conversational turn. This way, participants can compare them and recognize components they like more than others. Help users recognize ways to improve the chatbot’s responses by showing alternative chatbot responses. D.2 One piece of feedback we got from participants was that writing principles involves a difficult two-step process of first (1) articulating one’s feedback on the model’s current output, and then (2) converting this feedback into a principle for the LLM to follow. Often, one’s initial reaction to the model’s output is intuitive, and converting that intuition into a principle for the chatbot to follow can be challenging. In addition, once participants had a particular bit of feedback in mind (e.g., “I don’t like how the chatbot didn’t introduce itself”), they were unsure how to phrase their principle. However, in line with prior research [45, 46], they found that more concrete principles that specified what should happen and when (e.g., “Introduce yourself at the start of the conversation, and state what you can help with”) generally led to better results. Thus, our second design goal was to help users go from their initial reaction to the model’s output to a specific, clearly written principle to steer the model. Help convert user feedback into specific principles to make principle writing easier. D.3 As participants wrote more principles, they wanted ways to test these principles to make sure they worked. The early version of ConstitutionMaker only let users restart the conversation, and did not let users enable or disable principles. Users wanted to test individual principles on certain portions of the conversation, to see if the model was generating the correct content. And so, our last design goal was to enable easier testing of principles. Enable easier testing of principles to help users understand how well their principles are steering the chatbot’s behavior. 3.2 Principle Classification From the formative workshop and follow up sessions, we collected 79 principles in total and classified them to understand the kinds of principles users wanted to write. These principles correspond to a number of very different chatbots, including a show recommender, chemistry tutor, role playing game manager, travel agent and more. We describe common types of principles below. Unconditional principles are those that apply at every conversational turn. Examples include: (1) those that define a consistent personality of the bot (e.g., “Act grumpy all the time” or “Speak informally and in the first person”), (2) those that place guardrails on the conversational content (e.g., “Don’t talk about anything but planning a vacation”), and (3) those that establish a consistent form for the bot’s responses (e.g., “Limit responses to 20 words”). Meanwhile, a conditional principle only applies when a certain condition is meant. For example, “Generate an itinerary after all the information has been collected,” only applies to the conversation when some set of information has been acquired. Writing a conditional principle essentially defines a computational interaction; users establish a set of criteria that make the principle applicable to the conversation, and once that set of criteria is met, the principle is executed (e.g., an itinerary is generated). Principles can be either unconditional or conditional. For example, “Generate an itinerary after all the information has been collected” depends on the entire conversation history to determine if all of the requisite information has been collected. Similarly the following principle written for a machine learning tutor, “After verifying a user’s goal, provide help to solve their problem,” depends on the conversation history to identify if the user’s goal has been verified. Meanwhile, the principle “When the user says they had a particular cuisine the night before, recommend a different cuisine,” written for a food recommender, pertains just to the latest response by the user. Finally, the condition can depend on the action the bot is about to take, like “When providing a list of suggestions, use free text rather than bullet points,” which applies to any situation when the bot thinks it is appropriate to make suggestions. Conditional principles can depend on the entire conversation history, the user’s latest response, or the action the bot is about to take. For example, the principle “At the start of the conversation, introduce yourself and ask a fun question to kick off the conversation” is fulfilled in a single conversational turn, in which the bot introduces itself. Similarly, “Before recommending a restaurant, ask the user for their location” is also fulfilled in a single turn. Meanwhile, for a role playing game (RPG) bot that guides the user through an adventure, a participant wrote the following principle: “When the user tries to do something, put up small obstacles. Don’t let them succeed on the first attempt.” This principle implies that the bot needs to take action multiple turns prior to being fulfilled (e.g., by first putting up a small obstacle and then subsequently letting the user succeed). Similarly, for a travel agent bot, a user wrote “Ask questions one-by-one to get an idea of their preferences,” which also requires multiple conversational turns prior to fulfillment. Conditional principles can be fulfilled in a single or multiple conversational turns. In summary, principles can either be conditional, where they apply when a certain condition is met, or unconditional, where they apply at every conversational step. Conditional principles further break down into those that depend on the entire conversation history, the user’s last response, or the action the bot is about to take. And finally, conditional principles are either fulfilled in a single turn or multiple conversational turns. This paper is under CC 4.0 license. available on arxiv

Formative Study Illuminates the Path to Intuitive Chatbot Customization

A very warm welcome to another Company of the Week feature! Every week, we feature an awesome tech brand from our tech company database, making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. This week, we present our business blogging partner, Bybit- one of the top cryptocurrency exchanges offering tailored crypto services and product solutions for retail and institutional traders alike. Trusted by millions worldwide, Bybit made its name in a very short period.Here's the backstory of the inception and exceptional growth of Bybit byBen Zhou, Founder and CEO:"Bybit's inception was fueled by the team's recognition of the vast potential cryptocurrencies and blockchain technology offers to the finance ecosystem. Zhou talks about how they identified common issues faced by crypto traders, including inefficiencies in user experiences, liquidity constraints, and insufficient customer support. In the early stages of the business, Bybit faced significant challenges but unwavering determination and passion propelled them forward. They put forth their efforts towards building a robust infrastructure capable of accommodating high volumes of trading whilst prioritizing the security of client assets. These formative experiences laid the groundwork for Bybit's success and are etched in the company's history." Meet Bybit: Featured Business Stories Bybit has published 55 stories via HackerNoon Business Blogging Program, generating 1 month, 14 days, 2 hours, and 56 minutes of reading time so far! Here are our top 3 favorite stories by Bybit: Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained
How to Create and Sell NFTs: The Essential Guide
Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For That’s all this week! You can read the past Company of the Week newsletters here. A very warm welcome to another Company of the Week feature! Every week, we feature an awesome tech brand from our tech company database , making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. Company of the Week Company of the Week tech company database tech company database This week, we present our business blogging partner, Bybit - one of the top cryptocurrency exchanges offering tailored crypto services and product solutions for retail and institutional traders alike. Trusted by millions worldwide, Bybit made its name in a very short period. Here's the backstory of the inception and exceptional growth of Bybit by Ben Zhou, Founder and CEO : "Bybit's inception was fueled by the team's recognition of the vast potential cryptocurrencies and blockchain technology offers to the finance ecosystem. Zhou talks about how they identified common issues faced by crypto traders, including inefficiencies in user experiences, liquidity constraints, and insufficient customer support. In the early stages of the business, Bybit faced significant challenges but unwavering determination and passion propelled them forward. They put forth their efforts towards building a robust infrastructure capable of accommodating high volumes of trading whilst prioritizing the security of client assets. These formative experiences laid the groundwork for Bybit's success and are etched in the company's history." Bybit Bybit Ben Zhou, Founder and CEO Ben Zhou, Founder and CEO "Bybit's inception was fueled by the team's recognition of the vast potential cryptocurrencies and blockchain technology offers to the finance ecosystem. Zhou talks about how they identified common issues faced by crypto traders, including inefficiencies in user experiences, liquidity constraints, and insufficient customer support. In the early stages of the business, Bybit faced significant challenges but unwavering determination and passion propelled them forward. They put forth their efforts towards building a robust infrastructure capable of accommodating high volumes of trading whilst prioritizing the security of client assets. These formative experiences laid the groundwork for Bybit's success and are etched in the company's history." Meet Bybit: Featured Business Stories Bybit has published 55 stories via HackerNoon Business Blogging Program , generating 1 month, 14 days, 2 hours, and 56 minutes of reading time so far! Here are our top 3 favorite stories by Bybit: Bybit Bybit HackerNoon Business Blogging Program HackerNoon Business Blogging Program Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained How to Create and Sell NFTs: The Essential Guide Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained Crypto Chart Patterns: Price Continuation, Neutral, Reversal, And Targets Explained How to Create and Sell NFTs: The Essential Guide How to Create and Sell NFTs: The Essential Guide How to Create and Sell NFTs: The Essential Guide Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For Yes, This Is the Very Simple Guide to Cryptocurrencies You Were Looking For That’s all this week! You can read the past Company of the Week newsletters here . here

Meet Bybit: HackerNoon Company of the Week

মিট বাইবিট: সপ্তাহের হ্যাকারনুন কোম্পানি

Lernen Sie Bybit kennen: HackerNoon-Unternehmen der Woche

Conozca a Bybit: la empresa de la semana de HackerNoon

Rencontrez Bybit : entreprise HackerNoon de la semaine

बायबिट से मिलिए: हैकरनून कंपनी ऑफ द वीक

Bybit のご紹介: HackerNoon の今週の企業

Bybit을 만나보세요: 금주의 HackerNoon 회사

Conheça Bybit: Empresa HackerNoon da Semana

Встречайте Bybit: компания недели HackerNoon

Bybit ile Tanışın: Haftanın HackerNoon Şirketi

Gặp gỡ Bybit: Công ty HackerNoon của tuần

认识 Bybit：HackerNoon 本周公司

10 Questions with Founder and CEO Brian Vallelunga at Doppler via [The Startup Founder Interview Template]. 1. What is your company in 2–5 words? Scalable secrets management for developers. 2. Why is now the time for your company to exist? Developers are increasingly dependent on numerous services, each requiring authentication through API keys, database URLs, and other sensitive secrets. Secure management of these secrets is crucial, not only to safeguard a company's reputation but more importantly, to protect the personal data of potentially millions of users from breaches. As the complexity of digital services grows, the need for robust, streamlined solutions to secure and manage these secrets becomes more pressing. Our company exists to address this vital need, ensuring that security keeps pace with innovation and protecting all stakeholders involved. 3. What do you love about your team, and why are you the ones to solve this problem? It's often said, but in our case, it genuinely applies—our team is truly unique. What sets us apart is not just the expertise or the skill each member brings, but their extraordinary passion for our mission. This passion isn't just about the work; it's about making a meaningful impact. This shared commitment has fostered one of the most cohesive and vibrant cultures I've ever had the pleasure of being part of. It's this combination of passion, culture, and talent that uniquely positions us to tackle such challenging problems effectively. We aren't just working; we're driven by a collective mission to innovate and lead in our field, ensuring we provide the best solutions. 4. If you weren’t building your startup, what would you be doing? If I weren't dedicated to building our startup, I would likely be involved in another venture within the hard tech space, particularly one that leverages artificial intelligence to accelerate research and development. AI has the potential to vastly improve efficiency and innovation across various industries, and I'm deeply interested in exploring how these technologies can solve complex problems and drive progress. 5. At the moment, how do you measure success? What are your metrics? We operate our company based on a well-defined funnel, where each stage is guided by specific input and output KPIs, each assigned to a dedicated owner. Teams responsible for a shared stage in the funnel—both input and output KPIs—are required to hold weekly meetings to continuously refine and improve their performance metrics. At the top of the funnel, we focus on awareness KPIs, such as website visits, while at the bottom, we measure success through financial-based KPIs like the Annual Recurring Revenue (ARR) from self-serve and managed sales. To set our financial objectives, we begin with annual revenue targets established by the board and CEO, which our finance team then breaks down into quarterly goals. These are further divided into monthly KPIs by reversing through the pipeline: from revenue to awareness. These monthly KPIs are reviewed weekly in meetings with the leadership team and the KPI owners to ensure alignment and prompt action on emerging challenges and opportunities. 6. In a few sentences, what do you offer to whom? We provide a top-tier, developer-centric secrets management solution tailored for medium to large-scale companies. Our platform equips engineering teams with robust tools to store, manage, and orchestrate access to their secrets across various projects, teams, and environments, enhancing both security and operational efficiency. 7. What’s most exciting about your traction to date? The most thrilling aspect of our progress is our upward trajectory into larger markets. Our core mission is to secure the world's secrets, and it is particularly the larger enterprises that hold the highest density of sensitive information worth protecting. The recognition and adoption of our platform by these substantial institutions is not only a testament to the value we provide but also highlights the crucial need for our services in safeguarding their extensive digital assets. This shift is both validating and motivating as we continue to expand our reach and impact. 8. Where do you think your growth will be next year? While I am cautious about making speculative forecasts far in advance, if pressed for an estimate, I anticipate seeing our revenue grow by approximately 2 to 4x next year. 9. Tell us about your first paying customer and revenue expectations over the next year. Our very first paying customer was Tonebase! I had the opportunity to meet the founders through mutual friends and introduced them to our product with the Chipotle Sales Method. Their support was invaluable during our formative days—they provided consistent, high-quality feedback that was instrumental in shaping the foundational aspects of our product. Their early endorsement not only boosted our confidence but also set a strong precedent for the level of engagement we strive to maintain with all our customers. As for the upcoming year, we are optimistic about our revenue growth, aiming to build on this strong base and expand our reach further. 10. What’s your biggest threat? I love this question! As a security company, our most significant threat is undoubtedly the possibility of being hacked. We dedicate an immense amount of time, resources, and energy to ensure this does not happen, going above and beyond standard security practices. Despite these rigorous efforts, the risk remains—not just for us, but also for our customers. We believe in transparency and talk openly about the proactive steps we take to mitigate potential risks. Get Interviewed by HackerNoon via the Startup Founder Template 10 Questions with Founder and CEO Brian Vallelunga at Doppler via [ The Startup Founder Interview Template ]. Brian Vallelunga Doppler The Startup Founder Interview Template 1. What is your company in 2–5 words? Scalable secrets management for developers. 2. Why is now the time for your company to exist? Developers are increasingly dependent on numerous services, each requiring authentication through API keys, database URLs, and other sensitive secrets. Secure management of these secrets is crucial, not only to safeguard a company's reputation but more importantly, to protect the personal data of potentially millions of users from breaches. As the complexity of digital services grows, the need for robust, streamlined solutions to secure and manage these secrets becomes more pressing. Our company exists to address this vital need, ensuring that security keeps pace with innovation and protecting all stakeholders involved. 3. What do you love about your team, and why are you the ones to solve this problem? It's often said, but in our case, it genuinely applies—our team is truly unique. What sets us apart is not just the expertise or the skill each member brings, but their extraordinary passion for our mission. This passion isn't just about the work; it's about making a meaningful impact. This shared commitment has fostered one of the most cohesive and vibrant cultures I've ever had the pleasure of being part of. It's this combination of passion, culture, and talent that uniquely positions us to tackle such challenging problems effectively. We aren't just working; we're driven by a collective mission to innovate and lead in our field, ensuring we provide the best solutions. 4. If you weren’t building your startup, what would you be doing? If I weren't dedicated to building our startup, I would likely be involved in another venture within the hard tech space, particularly one that leverages artificial intelligence to accelerate research and development. AI has the potential to vastly improve efficiency and innovation across various industries, and I'm deeply interested in exploring how these technologies can solve complex problems and drive progress. 5. At the moment, how do you measure success? What are your metrics? We operate our company based on a well-defined funnel, where each stage is guided by specific input and output KPIs, each assigned to a dedicated owner. Teams responsible for a shared stage in the funnel—both input and output KPIs—are required to hold weekly meetings to continuously refine and improve their performance metrics. At the top of the funnel, we focus on awareness KPIs, such as website visits, while at the bottom, we measure success through financial-based KPIs like the Annual Recurring Revenue (ARR) from self-serve and managed sales. To set our financial objectives, we begin with annual revenue targets established by the board and CEO, which our finance team then breaks down into quarterly goals. These are further divided into monthly KPIs by reversing through the pipeline: from revenue to awareness. These monthly KPIs are reviewed weekly in meetings with the leadership team and the KPI owners to ensure alignment and prompt action on emerging challenges and opportunities. 6. In a few sentences, what do you offer to whom? We provide a top-tier, developer-centric secrets management solution tailored for medium to large-scale companies. Our platform equips engineering teams with robust tools to store, manage, and orchestrate access to their secrets across various projects, teams, and environments, enhancing both security and operational efficiency. 7. What’s most exciting about your traction to date? The most thrilling aspect of our progress is our upward trajectory into larger markets. Our core mission is to secure the world's secrets, and it is particularly the larger enterprises that hold the highest density of sensitive information worth protecting. The recognition and adoption of our platform by these substantial institutions is not only a testament to the value we provide but also highlights the crucial need for our services in safeguarding their extensive digital assets. This shift is both validating and motivating as we continue to expand our reach and impact. 8. Where do you think your growth will be next year? While I am cautious about making speculative forecasts far in advance, if pressed for an estimate, I anticipate seeing our revenue grow by approximately 2 to 4x next year. 9. Tell us about your first paying customer and revenue expectations over the next year. Our very first paying customer was Tonebase! I had the opportunity to meet the founders through mutual friends and introduced them to our product with the Chipotle Sales Method. Their support was invaluable during our formative days—they provided consistent, high-quality feedback that was instrumental in shaping the foundational aspects of our product. Their early endorsement not only boosted our confidence but also set a strong precedent for the level of engagement we strive to maintain with all our customers. As for the upcoming year, we are optimistic about our revenue growth, aiming to build on this strong base and expand our reach further. 10. What’s your biggest threat? I love this question! As a security company, our most significant threat is undoubtedly the possibility of being hacked. We dedicate an immense amount of time, resources, and energy to ensure this does not happen, going above and beyond standard security practices. Despite these rigorous efforts, the risk remains—not just for us, but also for our customers. We believe in transparency and talk openly about the proactive steps we take to mitigate potential risks. Get Interviewed by HackerNoon via the Startup Founder Template Get Interviewed by HackerNoon via the Startup Founder Template

Tech Founder Interview on Scalable Secrets Management for Developers and Scaling Yearly Revenue 2-4X

বিকাশকারীদের জন্য স্কেলেবল সিক্রেটস ম্যানেজমেন্ট এবং বার্ষিক আয় 2-4X স্কেল করার বিষয়ে টেক প্রতিষ্ঠাতা সাক্ষাৎকার

Interview mit Tech-Gründer zum Thema skalierbares Geheimnismanagement für Entwickler und Steigerung des Jahresumsatzes um das Zwei- bis Vierfache

Entrevista al fundador de tecnología sobre la gestión escalable de secretos para desarrolladores y el aumento de los ingresos anuales de 2 a 4 veces

Entretien avec le fondateur technique sur la gestion évolutive des secrets pour les développeurs et la multiplication par 2 à 4 des revenus annuels

डेवलपर्स के लिए स्केलेबल सीक्रेट्स मैनेजमेंट और वार्षिक राजस्व को 2-4X तक बढ़ाने पर टेक संस्थापक का साक्षात्कार

開発者向けのスケーラブルなシークレット管理と年間収益の 2 ～ 4 倍の拡大に関する技術創業者インタビュー

개발자를 위한 확장 가능한 비밀 관리 및 연간 수익 2~4배 확장에 대한 기술 창립자 인터뷰

Entrevista com o fundador da tecnologia sobre gerenciamento escalonável de segredos para desenvolvedores e aumento da receita anual de 2 a 4 vezes

Интервью с основателем технологии о масштабируемом управлении секретами для разработчиков и увеличении годового дохода в 2–4 раза

Geliştiriciler için Ölçeklenebilir Gizli Bilgi Yönetimi ve Yıllık Gelirin 2-4 Kat Arttırılması Konusunda Teknoloji Kurucusu Röportajı

Phỏng vấn người sáng lập công nghệ về quản lý bí mật có thể mở rộng dành cho nhà phát triển và mở rộng quy mô Doanh thu hàng năm gấp 2-4 lần

技术创始人访谈：面向开发人员的可扩展机密管理以及将年收入扩大 2-4 倍

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 5 Discussion The purpose of this study was to investigate the effects of a gamification element in a mathematical sciences course taught in flipped format. The study compared levels of engagement with pre-lecture quizzes before and after the gamification intervention was introduced. Comparisons were made across semesters (different courses, same students) and across academic years (same course, different students). Our study found that in both cases the level of engagement (measured by average percentage of students completing pre-lecture quizzes prior to the associated in-person sessions) was significantly higher when compared to a course where Level Up! was not implemented. Although engagement levels did drop over the course of each semester, they remained substantially higher for each week when compared to the previous semester or year. These results provide convincing evidence that introducing a gamification element to a course taught in flipped format increases students motivation and engagement to complete work outside of scheduled class time. When comparing engagement levels within students between two semesters, we found a general increase in engagement across both semesters but more interestingly found that some students with very low engagement levels in semester 1 went on the have some of the highest engagement levels in semester 2. For these students it seems that the gamification element drastically improved their motivation and engagement levels and is evidence that including a gamification element to a course may encourage engagement from students who would otherwise not engage. Our results also found that this increase in engagement could lead to an increase in performance with 82% of students receiving a grade of A, B or C in the semester 1 course of 2023-24 compared to 60% in 2022-23. Although we cannot say for sure what led to this increase in achievement and there are many factors which could influence this (such as cohort effect, different exam paper, etc.), an increase in engagement throughout a semester is compatible with an increase in performance since students are more likely to be distributing their learning over the course of a semester. This is especially beneficial for mathematical sciences courses where learning is often thought of as linear with each topic building on the previous one and poor understanding of early material can have a detrimental effect later in the course [Ireland and Mouthaan, 2020]. This study focuses on Year 1 undergraduate students who typically fall into a post-secondary education age range. The transition to university for students can be overwhelming, with changes in teaching delivery, workload and cognitive demands ([Blair, 2017]). Our results show that gamification can improve the learning experience, which may help with this transition, in line with observations by [Zaric et al., 2017]. This being said, [Kim and Castelli, 2021] looked at the effects of gamification across three age cohorts and found that university level participants found the least significant results in terms of positive interaction when compared to secondary school and adult learners, indicating that younger and older learners may have more interest in gamified factors. Currently, gamification is only present within the Year 1 modules of the undergraduate programme, and is not implemented within other modules on the programme. There is scope to explore the effects of gamification on a more mature cohort to observe if we see similar patterns of engagement. This study has considered student engagement levels in terms of completing pre-lecture quizzes, though it is important to note that we see meaningful engagement instead of cases where students are simply trying to ‘game’ the leaderboard. Our results are based solely on participation on formative reading quizzes where participation is identified through criteria being met, as this material was found to show that genuine participation took place. There are other activities in the course, such as textbook readings, which simply require a manual completion indicator and could be exposed to non-meaningful engagement. We feel that a reasonable performance on the quizzes indicates engagement with the other activities available (or indeed, of students independently finding their own resources for study). The results presented here are for a self-selected subset of students. However, course instructors were able to view results for the entire cohort of students to inform their future decision making on the learning and teaching approach for this course and others. The results presented here were found to be similar to the results for the entire cohort and the general conclusions made would not change if data on the entire cohort of students were considered. Although studies in gamification have reflected positive outcomes in terms of increased motivation and engagement in tasks, this teaching strategy has potential downsides. Effects such as increased competition between students ([Hakulinen et al., 2013]) and task evaluation difficulties, where certain tasks may not be suitable for a game based element or may have ways of being marked as complete without proper engagement in the learning materials ([Domínguez et al., 2013]) may become issues. The design of game based elements within a course must also be considered with a balance of reward and level of work and engagement required. Tasks which are overly complex can lead to a lack of engagement from learners ([Dong et al., 2012]). There is also potential for demotivation through the full gamification process, particulaly if students find themselves placed low on the points leaderboard. The competency levels of students may lead to negative consequences, including frustration and loss of self confidence ([Alomari et al., 2019]). A study by [Ding et al., 2018] showed that some students in a gamified learning activity may require more time to obtain the necessary understanding to progress through learning process. Engagement in gamification not only requires active participation from students, but time and attention from staff to implement the system effectively. From our experience using Level Up!, it takes time to set up the system. Several considerations have to be made at the set-up stage, including the allocation of experience points to certain tasks, and allocating points in such a way as to reflect the level of engagement required from the student. Levels were attainable for students which were split by weekly tasks, which requires manual intervention to tally the available points over a given week. For example, students could receive half points for completing certain tasks after a deadline which requires manual intervention from the course lecturer to reset the points awarded to a different total after each deadline. Though our study explored gamification using Level Up!, there are a host of other plug-ins which can be adapted in virtual learning environments such as Quizventure[4] and Block Game[5]. [Heilbrunn et al., 2017] provide a detailed review of tools for gamification analytics. Finally, we highlight the importance of using learning analytics, where possible, to access data from students on a course when measuring the success of a new teaching method or intervention. If the instructors of the two courses under consideration here had relied on feedback from students to inform teaching strategy - which was consistently positive even before the implementation of Level Up! - the true extent of the lack of engagement in the pre-lecture materials may never have been identified. This paper is available on arxiv under CC BY 4.0 DEED license. [4] https://github.com/xow/moodle-mod_quizgame [5] https://moodle.org/plugins/block_game Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 5 Discussion The purpose of this study was to investigate the effects of a gamification element in a mathematical sciences course taught in flipped format. The study compared levels of engagement with pre-lecture quizzes before and after the gamification intervention was introduced. Comparisons were made across semesters (different courses, same students) and across academic years (same course, different students). Our study found that in both cases the level of engagement (measured by average percentage of students completing pre-lecture quizzes prior to the associated in-person sessions) was significantly higher when compared to a course where Level Up! was not implemented. Although engagement levels did drop over the course of each semester, they remained substantially higher for each week when compared to the previous semester or year. These results provide convincing evidence that introducing a gamification element to a course taught in flipped format increases students motivation and engagement to complete work outside of scheduled class time. When comparing engagement levels within students between two semesters, we found a general increase in engagement across both semesters but more interestingly found that some students with very low engagement levels in semester 1 went on the have some of the highest engagement levels in semester 2. For these students it seems that the gamification element drastically improved their motivation and engagement levels and is evidence that including a gamification element to a course may encourage engagement from students who would otherwise not engage. Our results also found that this increase in engagement could lead to an increase in performance with 82% of students receiving a grade of A, B or C in the semester 1 course of 2023-24 compared to 60% in 2022-23. Although we cannot say for sure what led to this increase in achievement and there are many factors which could influence this (such as cohort effect, different exam paper, etc.), an increase in engagement throughout a semester is compatible with an increase in performance since students are more likely to be distributing their learning over the course of a semester. This is especially beneficial for mathematical sciences courses where learning is often thought of as linear with each topic building on the previous one and poor understanding of early material can have a detrimental effect later in the course [Ireland and Mouthaan, 2020]. This study focuses on Year 1 undergraduate students who typically fall into a post-secondary education age range. The transition to university for students can be overwhelming, with changes in teaching delivery, workload and cognitive demands ([Blair, 2017]). Our results show that gamification can improve the learning experience, which may help with this transition, in line with observations by [Zaric et al., 2017]. This being said, [Kim and Castelli, 2021] looked at the effects of gamification across three age cohorts and found that university level participants found the least significant results in terms of positive interaction when compared to secondary school and adult learners, indicating that younger and older learners may have more interest in gamified factors. Currently, gamification is only present within the Year 1 modules of the undergraduate programme, and is not implemented within other modules on the programme. There is scope to explore the effects of gamification on a more mature cohort to observe if we see similar patterns of engagement. This study has considered student engagement levels in terms of completing pre-lecture quizzes, though it is important to note that we see meaningful engagement instead of cases where students are simply trying to ‘game’ the leaderboard. Our results are based solely on participation on formative reading quizzes where participation is identified through criteria being met, as this material was found to show that genuine participation took place. There are other activities in the course, such as textbook readings, which simply require a manual completion indicator and could be exposed to non-meaningful engagement. We feel that a reasonable performance on the quizzes indicates engagement with the other activities available (or indeed, of students independently finding their own resources for study). The results presented here are for a self-selected subset of students. However, course instructors were able to view results for the entire cohort of students to inform their future decision making on the learning and teaching approach for this course and others. The results presented here were found to be similar to the results for the entire cohort and the general conclusions made would not change if data on the entire cohort of students were considered. Although studies in gamification have reflected positive outcomes in terms of increased motivation and engagement in tasks, this teaching strategy has potential downsides. Effects such as increased competition between students ([Hakulinen et al., 2013]) and task evaluation difficulties, where certain tasks may not be suitable for a game based element or may have ways of being marked as complete without proper engagement in the learning materials ([Domínguez et al., 2013]) may become issues. The design of game based elements within a course must also be considered with a balance of reward and level of work and engagement required. Tasks which are overly complex can lead to a lack of engagement from learners ([Dong et al., 2012]). There is also potential for demotivation through the full gamification process, particulaly if students find themselves placed low on the points leaderboard. The competency levels of students may lead to negative consequences, including frustration and loss of self confidence ([Alomari et al., 2019]). A study by [Ding et al., 2018] showed that some students in a gamified learning activity may require more time to obtain the necessary understanding to progress through learning process. Engagement in gamification not only requires active participation from students, but time and attention from staff to implement the system effectively. From our experience using Level Up!, it takes time to set up the system. Several considerations have to be made at the set-up stage, including the allocation of experience points to certain tasks, and allocating points in such a way as to reflect the level of engagement required from the student. Levels were attainable for students which were split by weekly tasks, which requires manual intervention to tally the available points over a given week. For example, students could receive half points for completing certain tasks after a deadline which requires manual intervention from the course lecturer to reset the points awarded to a different total after each deadline. Though our study explored gamification using Level Up!, there are a host of other plug-ins which can be adapted in virtual learning environments such as Quizventure[4] and Block Game[5]. [Heilbrunn et al., 2017] provide a detailed review of tools for gamification analytics. Finally, we highlight the importance of using learning analytics, where possible, to access data from students on a course when measuring the success of a new teaching method or intervention. If the instructors of the two courses under consideration here had relied on feedback from students to inform teaching strategy - which was consistently positive even before the implementation of Level Up! - the true extent of the lack of engagement in the pre-lecture materials may never have been identified. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [4] https://github.com/xow/moodle-mod_quizgame [5] https://moodle.org/plugins/block_game

Levelling Up Learning: Exploring the Impact of Gamification in Flipped Classrooms: Discussion

Levelling Up Learning: Exploring the Impact of Gamification in Flipped Classrooms: Findings

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 3 Motivation and intervention The pre-lecture activities were made available to students via the University of Glasgow’s virtual learning environment Moodle [2]. As such, it is straightforward for course instructors to check if students are engaging with these activities and crucially, given the flipped learning approach, whether they are engaging with them in advance of the in-person sessions. As discussed in Section 2.1, the University of Glasgow Learning Analytics Policy allows staff to undertake learning analytics activities which enable interventions to assist students learning. From the year in which the flipped learning approach was introduced (2020-21) staff on the course were therefore monitoring engagement in pre-lecture activities through activity logs on Moodle. For the purpose of this study, only engagement with the formative reading quizzes (pre-lecture quizzes) is measured, where activity completion can be indicated by certain conditions being met (e.g. whether a student has received a grade for a quiz), rather than students being able to manually mark completion (e.g. for link to reading from course textbook). This means that we cannot measure and compare engagement across all pre-lecture activities and we may be misrepresenting true engagement as some students may choose to read the textbook but not complete the reading quiz (or vice versa). However, this should give a more reliable estimate of meaningful engagement in activities since students have to engage somewhat with the activity (e.g. start and submit the quiz) rather than being able to manually mark that the activity is complete by checking a button. This does not fully guard against non-meaningful engagement (e.g. a student could begin the quiz, complete no questions, submit, and still be counted as having completed this activity) but performance in these activities (e.g. questions attempted and marks achieved) was also checked to understand the extent of meaningful engagement. Despite students indicating that they enjoyed the flipped learning approach through course evaluation forms, when looking at the data it was clear that engagement in the pre-lecture quizzes was extremely low. For example, the average completion of pre-lecture quizzes in advance of the active learning session taking place was 6.88% in semester 1 of 2022-23. This was of concern to course instructors given the in-class sessions involved spending less time presenting material and more time engaging in active learning activities as per the goal of flipped learning. This pattern was persistent over a number of years. Given that the introduction of the new courses coincided with the Covid-19 pandemic, it was also difficult to disentangle the lack of engagement with the flipped learning material, and the general lack of engagement HE institutions saw over the course of the pandemic ([Wester et al., 2021]). The academic year of 2022-23 saw a return to campus for the majority of teaching but the issue of low engagement in pre-lecture activities persisted. This is demonstrated in Figure 1 (a) which shows the cumulative mean percentage of students engaging in the pre-lecture quizzes for each week across an 11 week period in semester 1 of 2022-23. Each coloured line represents the cumulative average percentage of students who have engaged in the collection of quizzes associated with one week of teaching over the course of the semester (an average is shown here since each week may have a number of quizzes for students to complete). Of most interest is the level of engagement in the pre-lecture quizzes for the week in which the material is covered in class, which is represented by the first point for each weekly line in Figure 1 (a). These figures are also presented in column 1 of Table 1. This figure starts off low (around 27%) for week 1 and continues to drop for each week of material, before reaching a point around week 7 where less than 5% of students have engaged with the pre-lecture quizzes on the week in which the material is covered in class. Another interesting feature of Figure 1 (a) is the pattern of engagement for each collection of weekly quizzes over the semester. Although the engagement in the weekly material does increase slightly over the semester, at most only about half of the students in the class have ever engaged with the material at all on average for week 1, and after week 8 this drops to less than 25%. Interestingly, for all weeks an increase in average engagement can be seen between weeks 10 and 11 and we believe this to be related to exam preparation as the exam takes place immediately after the course finishes. These features are clearly of huge concern to the course instructors as they indicate that not only are students not engaging with the pre-lecture quizzes ‘on time’ to benefit from a flipped approach, the majority of students aren’t engaging with the materials at all across the entire duration of the course. In an attempt to tackle this low engagement the decision was made to add an element of gamification to the course which would encourage students to engage in the pre-lecture activities. This was implemented through the the Level Up! Moodle plug-in[3] which was introduced to the course in semester 2 of 2022-23. Level Up! allows students to gain points by participating in the course via Moodle. Points were awarded for the completion for a variety of course activities including the pre-lecture activities, tutorial questions, lab materials, and assessed quizzes and points were awarded based on completion only, i.e. the points system was endeavour-based and not merit-based. In an attempt to explicitly tackle engagement in the pre-lecture activities, full points were only awarded if the activity was completed before the associated lecture/tutorial/lab with half-points being awarded thereafter. Figure 2 shows how this plug-in is integrated within the course Moodle page with the left-hand image showing the Level Up! block that students can see on the main course page and the right-hand image showing the leaderboard view which shows students where they rank individually on an anonymous leaderboard throughout the semester. There is also an option to view where the students’ tutorial group rank in a team leaderboard. Level Up! was initially introduced in an attempt to increase engagement in pre-lecture activities in semester 2 of the 2022-23 academic year and, after initial success, introduced from semester 1 of the 2023-24 academic year. This allows us to make a number of comparisons which form our questions of interest: How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses.


How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. This paper is available on arxiv under CC BY 4.0 DEED license. [2] https://moodle.org/ [3] https://moodle.org/plugins/block_xp Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 3 Motivation and intervention The pre-lecture activities were made available to students via the University of Glasgow’s virtual learning environment Moodle [2]. As such, it is straightforward for course instructors to check if students are engaging with these activities and crucially, given the flipped learning approach, whether they are engaging with them in advance of the in-person sessions. As discussed in Section 2.1, the University of Glasgow Learning Analytics Policy allows staff to undertake learning analytics activities which enable interventions to assist students learning. From the year in which the flipped learning approach was introduced (2020-21) staff on the course were therefore monitoring engagement in pre-lecture activities through activity logs on Moodle. For the purpose of this study, only engagement with the formative reading quizzes (pre-lecture quizzes) is measured, where activity completion can be indicated by certain conditions being met (e.g. whether a student has received a grade for a quiz), rather than students being able to manually mark completion (e.g. for link to reading from course textbook). This means that we cannot measure and compare engagement across all pre-lecture activities and we may be misrepresenting true engagement as some students may choose to read the textbook but not complete the reading quiz (or vice versa). However, this should give a more reliable estimate of meaningful engagement in activities since students have to engage somewhat with the activity (e.g. start and submit the quiz) rather than being able to manually mark that the activity is complete by checking a button. This does not fully guard against non-meaningful engagement (e.g. a student could begin the quiz, complete no questions, submit, and still be counted as having completed this activity) but performance in these activities (e.g. questions attempted and marks achieved) was also checked to understand the extent of meaningful engagement. Despite students indicating that they enjoyed the flipped learning approach through course evaluation forms, when looking at the data it was clear that engagement in the pre-lecture quizzes was extremely low. For example, the average completion of pre-lecture quizzes in advance of the active learning session taking place was 6.88% in semester 1 of 2022-23. This was of concern to course instructors given the in-class sessions involved spending less time presenting material and more time engaging in active learning activities as per the goal of flipped learning. This pattern was persistent over a number of years. Given that the introduction of the new courses coincided with the Covid-19 pandemic, it was also difficult to disentangle the lack of engagement with the flipped learning material, and the general lack of engagement HE institutions saw over the course of the pandemic ([Wester et al., 2021]). The academic year of 2022-23 saw a return to campus for the majority of teaching but the issue of low engagement in pre-lecture activities persisted. This is demonstrated in Figure 1 (a) which shows the cumulative mean percentage of students engaging in the pre-lecture quizzes for each week across an 11 week period in semester 1 of 2022-23. Each coloured line represents the cumulative average percentage of students who have engaged in the collection of quizzes associated with one week of teaching over the course of the semester (an average is shown here since each week may have a number of quizzes for students to complete). Of most interest is the level of engagement in the pre-lecture quizzes for the week in which the material is covered in class, which is represented by the first point for each weekly line in Figure 1 (a). These figures are also presented in column 1 of Table 1. This figure starts off low (around 27%) for week 1 and continues to drop for each week of material, before reaching a point around week 7 where less than 5% of students have engaged with the pre-lecture quizzes on the week in which the material is covered in class. Another interesting feature of Figure 1 (a) is the pattern of engagement for each collection of weekly quizzes over the semester. Although the engagement in the weekly material does increase slightly over the semester, at most only about half of the students in the class have ever engaged with the material at all on average for week 1, and after week 8 this drops to less than 25%. Interestingly, for all weeks an increase in average engagement can be seen between weeks 10 and 11 and we believe this to be related to exam preparation as the exam takes place immediately after the course finishes. These features are clearly of huge concern to the course instructors as they indicate that not only are students not engaging with the pre-lecture quizzes ‘on time’ to benefit from a flipped approach, the majority of students aren’t engaging with the materials at all across the entire duration of the course. In an attempt to tackle this low engagement the decision was made to add an element of gamification to the course which would encourage students to engage in the pre-lecture activities. This was implemented through the the Level Up! Moodle plug-in[3] which was introduced to the course in semester 2 of 2022-23. Level Up! allows students to gain points by participating in the course via Moodle. Points were awarded for the completion for a variety of course activities including the pre-lecture activities, tutorial questions, lab materials, and assessed quizzes and points were awarded based on completion only, i.e. the points system was endeavour-based and not merit-based. In an attempt to explicitly tackle engagement in the pre-lecture activities, full points were only awarded if the activity was completed before the associated lecture/tutorial/lab with half-points being awarded thereafter. Figure 2 shows how this plug-in is integrated within the course Moodle page with the left-hand image showing the Level Up! block that students can see on the main course page and the right-hand image showing the leaderboard view which shows students where they rank individually on an anonymous leaderboard throughout the semester. There is also an option to view where the students’ tutorial group rank in a team leaderboard. Level Up! was initially introduced in an attempt to increase engagement in pre-lecture activities in semester 2 of the 2022-23 academic year and, after initial success, introduced from semester 1 of the 2023-24 academic year. This allows us to make a number of comparisons which form our questions of interest: How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! between the same students across two courses. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. How do engagement levels in pre-lecture quizzes compare after the implementation of Level Up! within the same course across two academic years. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [2] https://moodle.org/ [3] https://moodle.org/plugins/block_xp

Exploring the Impact of Gamification in Flipped Classrooms: Motivation and intervention

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 2 Course information At the University of Glasgow, the School of Mathematics and Statistics offer courses in introductory Statistics to undergraduate students across the University. These level 1 courses are typically taken in year 1 or 2 (pre-honours) of a student’s university career, which typically lasts 4 years. In year 1, single honours students will take courses in three subjects, one of which will be the subject they have chosen to specialise in. Any of these three courses can be considered in year 2, and students have the option to switch their chosen degree programme to any of these subjects conditional on meeting the requirements for progression. The purpose of the level 1 statistics courses is predominately to prepare students who are interested in studying statistics at higher levels and, given that most students will have little experience of statistics when they arrive at university, introduce students to the subject with the hope that students will choose to continue studying statistics in future years. There are two level 1 courses taught over two semesters (semester 1: Sep-Dec and semester 2: Jan-Mar) with final exams taking place at the end of each semester. Given the growing awareness of the importance of statistics across many disciplines, the courses have become increasingly popular over the past few years with 200+ students enrolled. To align with the University of Glasgow Learning and Teaching strategy, the courses were re-designed for 2020-21 with the aim of evolving the approach to learning and teaching towards student-centered, active learning. The approach taken for these courses was to introduce a flipped learning model, with a variety of online learning materials for students to complete before live teaching sessions. In-person sessions were then used to consolidate learning through a variety of interactive activities for students to complete individually or in small groups. For the purpose of this study we will call these online learning materials, ‘pre-lecture activities’. 2.1 Data The data for this study are taken from Moodle activity logs. These can be used to view all student interactions with Moodle, including engagement with the activities of interest for the courses under consideration here. Data such as these are becoming increasingly available and the use of data in the support of student learning is commonly referred to as learning analytics ([Leitner et al., 2017]). At this institution, staff may undertake learning analytics activities which enable interventions to assist students through the analysis of their own data. However, when such projects are intended for external dissemination they also require ethical approval. This study falls into the latter category and ethical approval was sought and granted in advance of this study taking place (application number 300220006). The research ethics associated with this study required students to consent to their data being used as part of this project. Consent was requested during live teaching sessions and was granted for n = 100 (out of a total of 257) students in 2022-23 and n = 77 (out of a total of 280) students in 2023-24. Unfortunately we were unable to obtain retrospective ethical approval to access data from 2020-21 (when flipped learning was introduced) and 2021-22 due to the opt-in nature of the study. We can therefore only provide results and comparisons for engagement using the 2022-23 and 2023-24 cohorts. Students attended three in-person sessions a week, and before each of these were asked to complete the following pre-lecture activities: • Textbook readings - students were asked to read a section of the course textbook in advance of the associated in-person session. The purpose of the readings was to introduce students to the course material. Completion of these can only be tracked through students manually marking completion via activity tracking on Moodle. • Formative reading quizzes - students were asked to complete a short quiz associated with after reading the corresponding textbook section. The purpose of the quizzes was for students to consolidate their learning and test their understanding of the course material in advance of the corresponding in-person session. Information on attempts for all quizzes is automatically stored in Moodle. It is therefore possible to determine which students have completed a quiz, when they started/finished the quiz, and their performance on the quiz. We can therefore track engagement in the quizzes and compare this with different cohorts as well as monitoring performance to inform the active learning sessions. These will be referred to as pre-lecture quizzes. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 2 Course information At the University of Glasgow, the School of Mathematics and Statistics offer courses in introductory Statistics to undergraduate students across the University. These level 1 courses are typically taken in year 1 or 2 (pre-honours) of a student’s university career, which typically lasts 4 years. In year 1, single honours students will take courses in three subjects, one of which will be the subject they have chosen to specialise in. Any of these three courses can be considered in year 2, and students have the option to switch their chosen degree programme to any of these subjects conditional on meeting the requirements for progression. The purpose of the level 1 statistics courses is predominately to prepare students who are interested in studying statistics at higher levels and, given that most students will have little experience of statistics when they arrive at university, introduce students to the subject with the hope that students will choose to continue studying statistics in future years. There are two level 1 courses taught over two semesters (semester 1: Sep-Dec and semester 2: Jan-Mar) with final exams taking place at the end of each semester. Given the growing awareness of the importance of statistics across many disciplines, the courses have become increasingly popular over the past few years with 200+ students enrolled. To align with the University of Glasgow Learning and Teaching strategy, the courses were re-designed for 2020-21 with the aim of evolving the approach to learning and teaching towards student-centered, active learning. The approach taken for these courses was to introduce a flipped learning model, with a variety of online learning materials for students to complete before live teaching sessions. In-person sessions were then used to consolidate learning through a variety of interactive activities for students to complete individually or in small groups. For the purpose of this study we will call these online learning materials, ‘pre-lecture activities’. 2.1 Data The data for this study are taken from Moodle activity logs. These can be used to view all student interactions with Moodle, including engagement with the activities of interest for the courses under consideration here. Data such as these are becoming increasingly available and the use of data in the support of student learning is commonly referred to as learning analytics ([Leitner et al., 2017]). At this institution, staff may undertake learning analytics activities which enable interventions to assist students through the analysis of their own data. However, when such projects are intended for external dissemination they also require ethical approval. This study falls into the latter category and ethical approval was sought and granted in advance of this study taking place (application number 300220006). The research ethics associated with this study required students to consent to their data being used as part of this project. Consent was requested during live teaching sessions and was granted for n = 100 (out of a total of 257) students in 2022-23 and n = 77 (out of a total of 280) students in 2023-24. Unfortunately we were unable to obtain retrospective ethical approval to access data from 2020-21 (when flipped learning was introduced) and 2021-22 due to the opt-in nature of the study. We can therefore only provide results and comparisons for engagement using the 2022-23 and 2023-24 cohorts. Students attended three in-person sessions a week, and before each of these were asked to complete the following pre-lecture activities: • Textbook readings - students were asked to read a section of the course textbook in advance of the associated in-person session. The purpose of the readings was to introduce students to the course material. Completion of these can only be tracked through students manually marking completion via activity tracking on Moodle. • Textbook readings • Formative reading quizzes - students were asked to complete a short quiz associated with after reading the corresponding textbook section. The purpose of the quizzes was for students to consolidate their learning and test their understanding of the course material in advance of the corresponding in-person session. Information on attempts for all quizzes is automatically stored in Moodle. It is therefore possible to determine which students have completed a quiz, when they started/finished the quiz, and their performance on the quiz. We can therefore track engagement in the quizzes and compare this with different cohorts as well as monitoring performance to inform the active learning sessions. These will be referred to as pre-lecture quizzes. • Formative reading quizzes This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Levelling Up Learning:Exploring the Impact of Gamification in Flipped Classrooms: Course information

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Conclusion Our analysis shows that while it may well be technically possible to design a search engine that manages to balance privacy-preservation with the promotion of high IQ material, this is currently not the case. The current relationship between privacy-preserving design features of search engines and the IQ of the results they return is inverse (although not proportionally). In instances where this can have a negative impact on public health, as in the example we have provided of the promotion of anti-vaccine misinformation, not intervening to alter the design of the algorithm even if this means sacrificing some degree of user privacy - can lead to severe harm for a large population of user and is, therefore, unethical. Designing a search engine that is privacy savvy and avoids issues with filter bubbles that can result from user-tracking may be a good thing, like designing a car with an engine that does not pollute and is inexpensive to run, and designers should seek to balance the different aspects of search engine design highlighted in Figure 3. However, if the brakes in an environmentally-designed car do not work, the car is unsafe and this negates the positive ethical decisions made by the designers. In a car this is a highly unlikely design outcome, as a car has to undergo several rounds of testing by regulatory agencies before being allowed on the market. This is not yet the case for search engines, which are only regulated from the perspective of data protection – which is primarily interpreted as data security rather than data privacy. Our study suggests that this is necessary but insufficient, and instead mechanisms should be developed to test search engines from the perspective of IQ, (particularly for YMYL webpages) before they can be deemed trustworthy providers of public health information. References 1. World Health Organization. Ten threats to global health in 2019. 2019. https://www.who.int/emergencies/ten-threats-to-global-health-in-2019 (archived at https://web.archive.org/web/20190627111040/https://www.who.int/emergencies/ten-threats-toglobal-health-in-2019) (accessed 27/06/2019 2. Wakefield AJ, Murch SH, Anthony A, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet 1998; 351(9103): 637-41. 3. Stahl JP, Cohen R, Denis F, et al. The impact of the web and social networks on vaccination. New challenges and opportunities offered to fight against vaccine hesitancy. Med Mal Infect 2016; 46(3): 117-22. 4. Martin GP, Sutton E, Willars J, Dixon-Woods M. Frameworks for change in healthcare organisations: a formative evaluation of the NHS Change Model. Health Serv Manage Res 2013; 26(2-3): 65-75. 5. Illari P, Floridi L. Information quality, data and philosophy. The Philosophy of Information Quality: Springer; 2014: 5-23. 6. Al-Jefri M, Evans R, Uchyigit G, Ghezzi P. What is health information quality? Ethical dimension and perception by users. Frontiers in medicine 2018; 5. 7. Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res 2006; 8(4): e27. 8. Chen X, Hay JL, Waters EA, et al. Health Literacy and Use and Trust in Health Information. J Health Commun 2018; 23(8): 724-34. 9. Paige SR, Krieger JL, Stellefson ML. The Influence of eHealth Literacy on Perceived Trust in Online Health Communication Channels and Sources. J Health Commun 2017; 22(1): 53-65. 10. Kim J, Park HA. Development of a health information technology acceptance model using consumers' health behavior intention. J Med Internet Res 2012; 14(5): e133. 11. Borah P, Xiao X. The Importance of 'Likes': The Interplay of Message Framing, Source, and Social Endorsement on Credibility Perceptions of Health Information on Facebook. J Health Commun 2018; 23(4): 399-411. 12. Huo C, Zhang M, Ma F. Factors influencing people’s health knowledge adoption in social media: the mediating effect of trust and the moderating effect of health threat. Library Hi Tech 2018; 36(1): 129-51. 13. Chiou L, Tucker C. Search engines and data retention: Implications for privacy and antitrust: National Bureau of Economic Research, 2017. 14. Esteve A. The business of personal data: Google, Facebook, and privacy issues in the EU and the USA. International Data Privacy Law 2017; 7(1): 36-47. 15. Floridi L, Taddeo M. What is data ethics? : The Royal Society; 2016. 16. Moskvitch K. Amazon’s Alexa deal with the NHS can help patients but at what cost? Wired, 2019. (accessed 8/11/2019). 17. Arif N, Al-Jefri M, Bizzi IH, et al. Fake News or Weak Science? Visibility and Characterization of Antivaccine Webpages Returned by Google in Different Languages and Countries. Front Immunol 2018; 9: 1215. 18. Maki A, Evans R, Ghezzi P. Bad news. Analysis of the quality of information on influenza prevention returned by Google in English and Italian. Frontiers in Immunology 2015; 6:616. doi: 10.3389/fimmu.2015.00616. 19. Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. Hoboken, NJ, USA: John Wiley & Sons; 2013. 20. Economist T. Seek and you shall find. Google rewards reputable reporting, not left-wing politics. The Economist 2019; 431(9146): 85. 21. Taddeo M, Floridi L. The Debate on the Moral Responsibilities of Online Service Providers. Science and engineering ethics 2015: 1-29. 22. Mittelstadt BD, Allo P, Taddeo M, Wachter S, Floridi L. The ethics of algorithms: Mapping the debate. Big Data & Society 2016; 3(2): 2053951716679679. 23. Association for Computing Machinery. ACM Code of Ethics and Professional Conduct. 2018. https://www.acm.org/code-of-ethics (archived at https://web.archive.org/web/20190627114921/https://www.acm.org/code-of-ethics) (accessed 27/06/2019 2019). 24. Tufekci Z. Algorithmic harms beyond Facebook and Google: Emergent challenges of computational agency. Colo Tech LJ 2015; 13: 203. 25. Hasnas J. Teaching business ethics: The principles approach. Journal of Business Ethics Education 2013; 10: 275-304. 26. Macdonald C, Gavura S. Alternative Medicine and the Ethics Of Commerce. Bioethics 2016; 30(2): 77-84. 27. Taddeo M, Floridi L. The Debate on the Moral Responsibilities of Online Service Providers. Sci Eng Ethics 2016; 22(6): 1575-603. 28. Chigwedere P, Seage GR, 3rd, Gruskin S, Lee TH, Essex M. Estimating the lost benefits of antiretroviral drug use in South Africa. J Acquir Immune Defic Syndr 2008; 49(4): 410-5. 29. Ouyang Y. Student's death highlights gaps in China's health regulations. Lancet Oncol 2016; 17(6): 709. 30. Browne M, Thomson P, Rockloff MJ, Pennycook G. Going against the Herd: Psychological and Cultural Factors Underlying the 'Vaccination Confidence Gap'. PLoS One 2015; 10(9): e0132562. 31. Bean SJ. Emerging and continuing trends in vaccine opposition website content. Vaccine 2011; 29(10): 1874-80.

Online Information of Vaccines: Conclusion and References

FORMOTIV

formotiv.com

-- employees

Since n.d.

COMPANY RANKING

FORMOTIV

EVERGREEN INDEX #4811

Representing this company? Click here to claim and customize this page!

ForMotiv WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

FORMOTIV

formotiv.com

-- employees

Since n.d.

COMPANY RANKING

FORMOTIV

EVERGREEN INDEX #4811

RELATED COMPANIESRANK

STORIES ABOUT FORMOTIV

Representing this company? Click here to claim and customize this page!

ForMotiv WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!