At Up Learn we use AI and cognitive science to ensure every student can achieve an A* regardless of their background

Up Learn

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 5 Discussion The purpose of this study was to investigate the effects of a gamification element in a mathematical sciences course taught in flipped format. The study compared levels of engagement with pre-lecture quizzes before and after the gamification intervention was introduced. Comparisons were made across semesters (different courses, same students) and across academic years (same course, different students). Our study found that in both cases the level of engagement (measured by average percentage of students completing pre-lecture quizzes prior to the associated in-person sessions) was significantly higher when compared to a course where Level Up! was not implemented. Although engagement levels did drop over the course of each semester, they remained substantially higher for each week when compared to the previous semester or year. These results provide convincing evidence that introducing a gamification element to a course taught in flipped format increases students motivation and engagement to complete work outside of scheduled class time. When comparing engagement levels within students between two semesters, we found a general increase in engagement across both semesters but more interestingly found that some students with very low engagement levels in semester 1 went on the have some of the highest engagement levels in semester 2. For these students it seems that the gamification element drastically improved their motivation and engagement levels and is evidence that including a gamification element to a course may encourage engagement from students who would otherwise not engage. Our results also found that this increase in engagement could lead to an increase in performance with 82% of students receiving a grade of A, B or C in the semester 1 course of 2023-24 compared to 60% in 2022-23. Although we cannot say for sure what led to this increase in achievement and there are many factors which could influence this (such as cohort effect, different exam paper, etc.), an increase in engagement throughout a semester is compatible with an increase in performance since students are more likely to be distributing their learning over the course of a semester. This is especially beneficial for mathematical sciences courses where learning is often thought of as linear with each topic building on the previous one and poor understanding of early material can have a detrimental effect later in the course [Ireland and Mouthaan, 2020]. This study focuses on Year 1 undergraduate students who typically fall into a post-secondary education age range. The transition to university for students can be overwhelming, with changes in teaching delivery, workload and cognitive demands ([Blair, 2017]). Our results show that gamification can improve the learning experience, which may help with this transition, in line with observations by [Zaric et al., 2017]. This being said, [Kim and Castelli, 2021] looked at the effects of gamification across three age cohorts and found that university level participants found the least significant results in terms of positive interaction when compared to secondary school and adult learners, indicating that younger and older learners may have more interest in gamified factors. Currently, gamification is only present within the Year 1 modules of the undergraduate programme, and is not implemented within other modules on the programme. There is scope to explore the effects of gamification on a more mature cohort to observe if we see similar patterns of engagement. This study has considered student engagement levels in terms of completing pre-lecture quizzes, though it is important to note that we see meaningful engagement instead of cases where students are simply trying to ‘game’ the leaderboard. Our results are based solely on participation on formative reading quizzes where participation is identified through criteria being met, as this material was found to show that genuine participation took place. There are other activities in the course, such as textbook readings, which simply require a manual completion indicator and could be exposed to non-meaningful engagement. We feel that a reasonable performance on the quizzes indicates engagement with the other activities available (or indeed, of students independently finding their own resources for study). The results presented here are for a self-selected subset of students. However, course instructors were able to view results for the entire cohort of students to inform their future decision making on the learning and teaching approach for this course and others. The results presented here were found to be similar to the results for the entire cohort and the general conclusions made would not change if data on the entire cohort of students were considered. Although studies in gamification have reflected positive outcomes in terms of increased motivation and engagement in tasks, this teaching strategy has potential downsides. Effects such as increased competition between students ([Hakulinen et al., 2013]) and task evaluation difficulties, where certain tasks may not be suitable for a game based element or may have ways of being marked as complete without proper engagement in the learning materials ([Domínguez et al., 2013]) may become issues. The design of game based elements within a course must also be considered with a balance of reward and level of work and engagement required. Tasks which are overly complex can lead to a lack of engagement from learners ([Dong et al., 2012]). There is also potential for demotivation through the full gamification process, particulaly if students find themselves placed low on the points leaderboard. The competency levels of students may lead to negative consequences, including frustration and loss of self confidence ([Alomari et al., 2019]). A study by [Ding et al., 2018] showed that some students in a gamified learning activity may require more time to obtain the necessary understanding to progress through learning process. Engagement in gamification not only requires active participation from students, but time and attention from staff to implement the system effectively. From our experience using Level Up!, it takes time to set up the system. Several considerations have to be made at the set-up stage, including the allocation of experience points to certain tasks, and allocating points in such a way as to reflect the level of engagement required from the student. Levels were attainable for students which were split by weekly tasks, which requires manual intervention to tally the available points over a given week. For example, students could receive half points for completing certain tasks after a deadline which requires manual intervention from the course lecturer to reset the points awarded to a different total after each deadline. Though our study explored gamification using Level Up!, there are a host of other plug-ins which can be adapted in virtual learning environments such as Quizventure[4] and Block Game[5]. [Heilbrunn et al., 2017] provide a detailed review of tools for gamification analytics. Finally, we highlight the importance of using learning analytics, where possible, to access data from students on a course when measuring the success of a new teaching method or intervention. If the instructors of the two courses under consideration here had relied on feedback from students to inform teaching strategy - which was consistently positive even before the implementation of Level Up! - the true extent of the lack of engagement in the pre-lecture materials may never have been identified. This paper is available on arxiv under CC BY 4.0 DEED license. [4] https://github.com/xow/moodle-mod_quizgame [5] https://moodle.org/plugins/block_game Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 5 Discussion The purpose of this study was to investigate the effects of a gamification element in a mathematical sciences course taught in flipped format. The study compared levels of engagement with pre-lecture quizzes before and after the gamification intervention was introduced. Comparisons were made across semesters (different courses, same students) and across academic years (same course, different students). Our study found that in both cases the level of engagement (measured by average percentage of students completing pre-lecture quizzes prior to the associated in-person sessions) was significantly higher when compared to a course where Level Up! was not implemented. Although engagement levels did drop over the course of each semester, they remained substantially higher for each week when compared to the previous semester or year. These results provide convincing evidence that introducing a gamification element to a course taught in flipped format increases students motivation and engagement to complete work outside of scheduled class time. When comparing engagement levels within students between two semesters, we found a general increase in engagement across both semesters but more interestingly found that some students with very low engagement levels in semester 1 went on the have some of the highest engagement levels in semester 2. For these students it seems that the gamification element drastically improved their motivation and engagement levels and is evidence that including a gamification element to a course may encourage engagement from students who would otherwise not engage. Our results also found that this increase in engagement could lead to an increase in performance with 82% of students receiving a grade of A, B or C in the semester 1 course of 2023-24 compared to 60% in 2022-23. Although we cannot say for sure what led to this increase in achievement and there are many factors which could influence this (such as cohort effect, different exam paper, etc.), an increase in engagement throughout a semester is compatible with an increase in performance since students are more likely to be distributing their learning over the course of a semester. This is especially beneficial for mathematical sciences courses where learning is often thought of as linear with each topic building on the previous one and poor understanding of early material can have a detrimental effect later in the course [Ireland and Mouthaan, 2020]. This study focuses on Year 1 undergraduate students who typically fall into a post-secondary education age range. The transition to university for students can be overwhelming, with changes in teaching delivery, workload and cognitive demands ([Blair, 2017]). Our results show that gamification can improve the learning experience, which may help with this transition, in line with observations by [Zaric et al., 2017]. This being said, [Kim and Castelli, 2021] looked at the effects of gamification across three age cohorts and found that university level participants found the least significant results in terms of positive interaction when compared to secondary school and adult learners, indicating that younger and older learners may have more interest in gamified factors. Currently, gamification is only present within the Year 1 modules of the undergraduate programme, and is not implemented within other modules on the programme. There is scope to explore the effects of gamification on a more mature cohort to observe if we see similar patterns of engagement. This study has considered student engagement levels in terms of completing pre-lecture quizzes, though it is important to note that we see meaningful engagement instead of cases where students are simply trying to ‘game’ the leaderboard. Our results are based solely on participation on formative reading quizzes where participation is identified through criteria being met, as this material was found to show that genuine participation took place. There are other activities in the course, such as textbook readings, which simply require a manual completion indicator and could be exposed to non-meaningful engagement. We feel that a reasonable performance on the quizzes indicates engagement with the other activities available (or indeed, of students independently finding their own resources for study). The results presented here are for a self-selected subset of students. However, course instructors were able to view results for the entire cohort of students to inform their future decision making on the learning and teaching approach for this course and others. The results presented here were found to be similar to the results for the entire cohort and the general conclusions made would not change if data on the entire cohort of students were considered. Although studies in gamification have reflected positive outcomes in terms of increased motivation and engagement in tasks, this teaching strategy has potential downsides. Effects such as increased competition between students ([Hakulinen et al., 2013]) and task evaluation difficulties, where certain tasks may not be suitable for a game based element or may have ways of being marked as complete without proper engagement in the learning materials ([Domínguez et al., 2013]) may become issues. The design of game based elements within a course must also be considered with a balance of reward and level of work and engagement required. Tasks which are overly complex can lead to a lack of engagement from learners ([Dong et al., 2012]). There is also potential for demotivation through the full gamification process, particulaly if students find themselves placed low on the points leaderboard. The competency levels of students may lead to negative consequences, including frustration and loss of self confidence ([Alomari et al., 2019]). A study by [Ding et al., 2018] showed that some students in a gamified learning activity may require more time to obtain the necessary understanding to progress through learning process. Engagement in gamification not only requires active participation from students, but time and attention from staff to implement the system effectively. From our experience using Level Up!, it takes time to set up the system. Several considerations have to be made at the set-up stage, including the allocation of experience points to certain tasks, and allocating points in such a way as to reflect the level of engagement required from the student. Levels were attainable for students which were split by weekly tasks, which requires manual intervention to tally the available points over a given week. For example, students could receive half points for completing certain tasks after a deadline which requires manual intervention from the course lecturer to reset the points awarded to a different total after each deadline. Though our study explored gamification using Level Up!, there are a host of other plug-ins which can be adapted in virtual learning environments such as Quizventure[4] and Block Game[5]. [Heilbrunn et al., 2017] provide a detailed review of tools for gamification analytics. Finally, we highlight the importance of using learning analytics, where possible, to access data from students on a course when measuring the success of a new teaching method or intervention. If the instructors of the two courses under consideration here had relied on feedback from students to inform teaching strategy - which was consistently positive even before the implementation of Level Up! - the true extent of the lack of engagement in the pre-lecture materials may never have been identified. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [4] https://github.com/xow/moodle-mod_quizgame [5] https://moodle.org/plugins/block_game

Levelling Up Learning: Exploring the Impact of Gamification in Flipped Classrooms: Discussion

Levelling Up Learning: Exploring the Impact of Gamification in Flipped Classrooms: Findings

Authors:
(1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk);
(2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk);
(3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction
Course information
Motivation and intervention
Findings
Discussion
Conclusion and References 2 Course information At the University of Glasgow, the School of Mathematics and Statistics offer courses in introductory Statistics to undergraduate students across the University. These level 1 courses are typically taken in year 1 or 2 (pre-honours) of a student’s university career, which typically lasts 4 years. In year 1, single honours students will take courses in three subjects, one of which will be the subject they have chosen to specialise in. Any of these three courses can be considered in year 2, and students have the option to switch their chosen degree programme to any of these subjects conditional on meeting the requirements for progression. The purpose of the level 1 statistics courses is predominately to prepare students who are interested in studying statistics at higher levels and, given that most students will have little experience of statistics when they arrive at university, introduce students to the subject with the hope that students will choose to continue studying statistics in future years. There are two level 1 courses taught over two semesters (semester 1: Sep-Dec and semester 2: Jan-Mar) with final exams taking place at the end of each semester. Given the growing awareness of the importance of statistics across many disciplines, the courses have become increasingly popular over the past few years with 200+ students enrolled. To align with the University of Glasgow Learning and Teaching strategy, the courses were re-designed for 2020-21 with the aim of evolving the approach to learning and teaching towards student-centered, active learning. The approach taken for these courses was to introduce a flipped learning model, with a variety of online learning materials for students to complete before live teaching sessions. In-person sessions were then used to consolidate learning through a variety of interactive activities for students to complete individually or in small groups. For the purpose of this study we will call these online learning materials, ‘pre-lecture activities’. 2.1 Data The data for this study are taken from Moodle activity logs. These can be used to view all student interactions with Moodle, including engagement with the activities of interest for the courses under consideration here. Data such as these are becoming increasingly available and the use of data in the support of student learning is commonly referred to as learning analytics ([Leitner et al., 2017]). At this institution, staff may undertake learning analytics activities which enable interventions to assist students through the analysis of their own data. However, when such projects are intended for external dissemination they also require ethical approval. This study falls into the latter category and ethical approval was sought and granted in advance of this study taking place (application number 300220006). The research ethics associated with this study required students to consent to their data being used as part of this project. Consent was requested during live teaching sessions and was granted for n = 100 (out of a total of 257) students in 2022-23 and n = 77 (out of a total of 280) students in 2023-24. Unfortunately we were unable to obtain retrospective ethical approval to access data from 2020-21 (when flipped learning was introduced) and 2021-22 due to the opt-in nature of the study. We can therefore only provide results and comparisons for engagement using the 2022-23 and 2023-24 cohorts. Students attended three in-person sessions a week, and before each of these were asked to complete the following pre-lecture activities: • Textbook readings - students were asked to read a section of the course textbook in advance of the associated in-person session. The purpose of the readings was to introduce students to the course material. Completion of these can only be tracked through students manually marking completion via activity tracking on Moodle. • Formative reading quizzes - students were asked to complete a short quiz associated with after reading the corresponding textbook section. The purpose of the quizzes was for students to consolidate their learning and test their understanding of the course material in advance of the corresponding in-person session. Information on attempts for all quizzes is automatically stored in Moodle. It is therefore possible to determine which students have completed a quiz, when they started/finished the quiz, and their performance on the quiz. We can therefore track engagement in the quizzes and compare this with different cohorts as well as monitoring performance to inform the active learning sessions. These will be referred to as pre-lecture quizzes. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Authors: Authors: (1) Eilidh Jack, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (eilidh.jack@glasgow.ac.uk); (2) Craig Alexander, School of Mathematics and Statistics, University of Glasgow, Glasgow, G12 8QQ (craig.alexander.2@glasgow.ac.uk); (3) Elinor Jones, Department of Statistical Science, University College London, London, WC1E 7HB (elinor.jones@ucl.ac.uk). Table of Links Abstract and Introduction Course information Motivation and intervention Findings Discussion Conclusion and References Abstract and Introduction Abstract and Introduction Course information Course information Motivation and intervention Motivation and intervention Findings Findings Discussion Discussion Conclusion and References Conclusion and References 2 Course information At the University of Glasgow, the School of Mathematics and Statistics offer courses in introductory Statistics to undergraduate students across the University. These level 1 courses are typically taken in year 1 or 2 (pre-honours) of a student’s university career, which typically lasts 4 years. In year 1, single honours students will take courses in three subjects, one of which will be the subject they have chosen to specialise in. Any of these three courses can be considered in year 2, and students have the option to switch their chosen degree programme to any of these subjects conditional on meeting the requirements for progression. The purpose of the level 1 statistics courses is predominately to prepare students who are interested in studying statistics at higher levels and, given that most students will have little experience of statistics when they arrive at university, introduce students to the subject with the hope that students will choose to continue studying statistics in future years. There are two level 1 courses taught over two semesters (semester 1: Sep-Dec and semester 2: Jan-Mar) with final exams taking place at the end of each semester. Given the growing awareness of the importance of statistics across many disciplines, the courses have become increasingly popular over the past few years with 200+ students enrolled. To align with the University of Glasgow Learning and Teaching strategy, the courses were re-designed for 2020-21 with the aim of evolving the approach to learning and teaching towards student-centered, active learning. The approach taken for these courses was to introduce a flipped learning model, with a variety of online learning materials for students to complete before live teaching sessions. In-person sessions were then used to consolidate learning through a variety of interactive activities for students to complete individually or in small groups. For the purpose of this study we will call these online learning materials, ‘pre-lecture activities’. 2.1 Data The data for this study are taken from Moodle activity logs. These can be used to view all student interactions with Moodle, including engagement with the activities of interest for the courses under consideration here. Data such as these are becoming increasingly available and the use of data in the support of student learning is commonly referred to as learning analytics ([Leitner et al., 2017]). At this institution, staff may undertake learning analytics activities which enable interventions to assist students through the analysis of their own data. However, when such projects are intended for external dissemination they also require ethical approval. This study falls into the latter category and ethical approval was sought and granted in advance of this study taking place (application number 300220006). The research ethics associated with this study required students to consent to their data being used as part of this project. Consent was requested during live teaching sessions and was granted for n = 100 (out of a total of 257) students in 2022-23 and n = 77 (out of a total of 280) students in 2023-24. Unfortunately we were unable to obtain retrospective ethical approval to access data from 2020-21 (when flipped learning was introduced) and 2021-22 due to the opt-in nature of the study. We can therefore only provide results and comparisons for engagement using the 2022-23 and 2023-24 cohorts. Students attended three in-person sessions a week, and before each of these were asked to complete the following pre-lecture activities: • Textbook readings - students were asked to read a section of the course textbook in advance of the associated in-person session. The purpose of the readings was to introduce students to the course material. Completion of these can only be tracked through students manually marking completion via activity tracking on Moodle. • Textbook readings • Formative reading quizzes - students were asked to complete a short quiz associated with after reading the corresponding textbook section. The purpose of the quizzes was for students to consolidate their learning and test their understanding of the course material in advance of the corresponding in-person session. Information on attempts for all quizzes is automatically stored in Moodle. It is therefore possible to determine which students have completed a quiz, when they started/finished the quiz, and their performance on the quiz. We can therefore track engagement in the quizzes and compare this with different cohorts as well as monitoring performance to inform the active learning sessions. These will be referred to as pre-lecture quizzes. • Formative reading quizzes This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Levelling Up Learning:Exploring the Impact of Gamification in Flipped Classrooms: Course information

I want to tell you about a fabulous community for working on goals! For the past year I have been working with an accountability and productivity community based in Slack and Notion called https://www.puddlepod.com/.  Puddle Pod is the creation of entrepreneur Michael Batko. When I signed up for the Australian group in 2023, it was primarily a productivity course, but a course on AI has been added as well. Per the website, “5-week virtual productivity course to simplify your life, spend time on what’s important and make you more effective at work.” I am based in the United States and found out about the group from an off hand mention by a business coach. Puddle Pod Even though I was in the United States and Puddle Pod is based a world away in Australia, I was still able to participate in the community to a good degree. I was able to participate in some group meetings and meet people primarily in Sydney and Melbourne. The group meetings were called Pods. We went through the productivity information on Notion and tried to apply these concepts and we met in our pod groups once a week to talk over our progress and homework. I found these meetings to be particularly helpful.  I tried to choose a group time at noon or in the morning Sydney time because I am in the Eastern standard time zone. I learned a lot from others in the group and I learned a lot from the Notion education materials. Everyone was friendly to me and I was very happy to be part of the group. Another aspect of the group that I particularly liked was Habit Pod. When I participated, Habit Pod was a Slack channel that you could post your habit goals in. Then in this channel, you could update your progress on your habit goals daily. There was another Puddle Pod classmate who posted daily so I was able to see how the Habit Pod worked. When you posted in the Habit Pod, you were supposed to post your goal and then a picture as sort of proof you had done that action.  Others might be able to support your habit progress through posting emojis for encouragement.  Using Habit Pod was particularly addictive for me because I had always wanted to work on my goals and I really enjoyed the accountability and the community. I initially posted inconsistently in Habit Pod, but sometime in the summer of 2023, I started posting my goals and habit progress regularly. The goals changed over time, but I posted everyday for about 1 year. What I noticed when I posted my goals in an accountability group was that I was very dedicated to completing them or at least getting them partially done. There were times when I would have to post that I had done nothing on my goal or that I was taking a break that day, but for the most part I felt very engaged in Habit Pod. I also could do Habit Pod more easily because I am in a different time zone than many of the others. Many people were really kind and put an encouraging emoji next to my progress. I found it joyful to help others and encourage them in the goals as well. The accountability group helped me and consistently posting my goals allowed me to reevaluate what my goals were.  It was like an evolution of goals!  I learned my values more clearly by doing this. I decided to change my goals after reading Celeste Headlee’s book entitled, “Do Nothing” to be goals that were more forgiving, self-compassionate, and more open.  For example, Instead of reading 20 pages of a book, my goal might be to “Learn Something New."  That allowed for more creative expression and I ended up learning more. I posted everyday for about 1 year. In the process, I learned a lot about myself because I took detours, changed goals and even failed a lot. I had the opportunity to visit Australia and meet Puddle Pod Founder Michael Batko during my experiences. This Puddle Pod experience was tremendous for me and I want to ensure you know about it. It can really help with your goal setting success! The next pod sign up is May 2024, but if that time does not work for you, they have a new start every quarter. Puddle Pod is recognized as a school on Linkedin.com so that means that your educational allowance from your workplace might cover it. Puddle Pod and its offerings are continually changing so I might not have the most up to date information. What I do know is that it's a tremendous educational experience.  You will get out what you put in and maybe even a ton more. Since I stopped posting in Habit Pod, I found an accountability solution in the United States, I am still consistently posting my habits and working on them! I found out something I did not know about myself and I feel I have been changed by the experience of meeting others in Puddle Pod and learning about myself through working on my goals.

A Year Long Adventure Working on Goals and Productivity

I led the growth team at Codecademy and helped the company grow from $10M in ARR to $50M ARR+ before it . sold to a company called Skillsoft for $525M last year I learned a lot about the details that make B2C SaaS/Recurring revenue business models work well that I have yet to see written down anywhere. My real motivation here is to help the companies that have achieved a good product market fit and have yet to figure out how to make money. To survive in the long term, you need to do both, and plenty of great products have died because they didn't build out the business side fast enough. These lessons directly apply to freemium/SaaS businesses and (probably?) apply to other recurring revenue businesses such as subscription e-commerce, agencies, consulting, etc. I plan to cover the following: The mistakes we made and lessons we learned The most effective levers we found at each layer of the business (Payments, Churn Reduction, Extending Lifecycles of Users, Pricing and Packaging, Trials/Paywalls) COMPANY CONTEXT Codecademy, for those unfamiliar, is a B2C tech education platform that focuses on taking total beginners and getting them to a medium level of technical skills. The company started in 2011 and built what I still think is the best free learn-to-code product online. As all "advice" really depends on the context that it's coming from, here is where we were at as a company in roughly 2017 : Very strong, low-cost free user acquisition driven by the free product, excellent brand name, and strong SEO ranking Product market fit in the free product for users who are early-stage learners (i.e., not professional software engineers) Short/Intermittent usage patterns across all users - E.g., they show up for a few weeks/months to learn something, then leave for a while, then maybe come back in a few months/years. Not the best use case, but a solid one A paid product that was built but was still being figured out. TL;DR - We had a lot of free users but were terrible at converting them to paid and monetizing them. The final bit of context is that I am a product manager, so most of our work was on the product rather than in marketing channels. We did have a great marketing team; however, I'll mainly talk about what I implemented firsthand and saw work. THINGS WE SCREWED UP Like most first-time growth teams, we made many of the classic mistakes in our first year. The most painful/memorable of those are: 1. Not understanding what to A/B test vs. just shipping it When we first tried to set up an A/B testing program, we got way too lost in the concept of "measuring" all of the impacts that we were having and not making any "assumptions". That was dumb. We actually debated if we should have a pricing page on the site and how we might measure that impact. I remember that adding a pricing page to the header had a small to medium-sized impact on conversion, which is great in theory... However, I would actually consider that one a loss, as we should have had this in the header the whole time. Don't debate the best practices, just implement them and figure our where you should break them. 2. Launching giant projects without enough due diligence The flip side of this, is that you should not assume that A/B testing is an excuse not to due the due diligence that you need on big projects. I think we shipped 3 different pricing page "layouts" to see which was more effective. I think that it took two months to run, was super expensive from a design and engineering perspective, blocked other tests and in the end, we didn't learn anything. We should have just made a decision. 3. Thinking of ideas that were "fun" or "fancy." One of the tenets of "Growth" is that you should be able to test any idea that you have a solid hypothesis behind. While I think this is valuable, its way more important to start with a solid foundation of best practices in your area. The "fancy" ideas (such as "running subway ads" or "partnering with influencers") only work if you have a solid funnel to drive this traffic into. 4. Working in the wrong areas of the application As I mentioned above, we re-designed our pricing page twice before realizing that a very small % of users purchase right from that page.  Most are purchased right from the product itself without going back to pricing. 5. Not iterating on good ideas that didn't work out the first time In retrospect, we moved on way too quickly from ideas that we had decent confidence in that didn't work the first time. If you are hitting a real customer problem, you did your research and V1 isn't working out, then I would take another shot. 6. Not implementing best practices fast enough As mentioned above, don't A/B test the basics.  Just ship them. 7. Not thinking through if a small win was worth the complexity that we're adding to the system & company operations. When you combine this with the friction of getting all the necessary tooling set up, learning how to run an A/B testing program, and building out the team.... this wasn't our best year. Additionally, B2C SaaS and subscription businesses can be tricky to grow for a few reasons, especially when paired with a free product/tier. These businesses can take a lot longer to grow because you aren't going to close a big enterprise contract, and scaling paid media is more challenging. The value you capture from users is spread across their lifecycle, so being confident about your LTV and the spending in marketing channels is hard (not impossible, but hard) Any time you do something like test new pricing (which you should), you can't really know the impact on LTV because those new users haven't reached the end of their lifecycle. When you're also running a big free product, it is hard to run direct response style paid media effectively as you can't really prevent users from finding the free product. Understanding payback periods for ad spend gets more challenging when they might spend variable amounts of time in the free tier based on the audience. OUR BIGGEST LESSONS Recurring revenue businesses are basically structured like the above diagram. Many companies go wrong because they only focus on making acquisition cheaper and faster when in reality, you should also have been systematically improving LTV to afford higher acquisition costs. There is a Dan Kennedy quote, which is something like: "The business that can pay the most to acquire the customer will win." Assuming that you don't have a bunch of VC money light on fire (and even if you do), the best chance that you have in the long term is figuring out to better monetize each monthly cohort of users and increase your overall LTV, which will allow you to spend more to acquire more customers profitably. Here is what I would advise companies in a similar place to do. This is a path toward reliably making improvements, not seeing giant lifts. 1. Figure out monetization early in your product/company's life. Everything gets more expensive to change the longer you wait. It involves convincing people, changing more code, retraining more sales/customer service people, etc. Every cohort that you under-monetize during your early years is money that you're never getting back. 2. While speed is essential in all businesses, it's extra important in subscription businesses Subscription businesses are slower to scale than most B2B businesses because most of your customers will pay incrementally across their lifecycle. You are not likely to close a big enterprise account that allows you to collect a lot of cash upfront. Because of this, every missed cohort of users that you didn't monetize well is a lost forever. The faster you get the best practices running, the more revenue you'll have to compound the following years. Don't overthink the fundamentals. 3. Work from the bottom of the funnel upwards unless you have a good reason not to. You're active paying users are hands down the most valuable to you. You don't want to lose them for things like payment failures that could have been prevented. If you are in a company that drives decisions based on A/B tests, you will be beholden to a concept called minimum detectable effect (MDE). MDE is, basically, the smallest lift you can measure with statistical significance (which basically translates to confidence). So if you have a high MDE, say 20%, and you ship a test that improves something by 19.99%, you won't be able to measure with statistical significance. There are ways of compensating for this, but this makes testing hard. The worse your overall conversion is, the larger the MDE will be, which makes A/B testing harder. But the upside is that improving things at the bottom of your funnel will lower (a good thing) the MDE, which raises the accuracy of your ability to measure things above it in the funnel. 4. It's hard to convince people to pay you money, so start by finding all the people who are trying to pay you, and it's not working Find all the dead ends and confusion across your product and fix those. Ensure the checkout page error states are clear, that any payment integrations aren't broken in specific browsers, etc. Are the bugs that happen during certain edge cases in important parts of the app? 5. Try to keep complexity low. Your goal is to keep development and operational speed fast, so check that any changes you make are worth the maintenance costs. For example, Suppose you find that setting up custom price structures per country raises revenue. From now on, the finance team will need to build multiple models to forecast revenue, the customer service now needs multiple sets of auto-responses, etc. 6. Focus where most users are and focus on improving clarity. Try to find the places in your product that most users are converting from and ensure that you're setting context correctly the step before. Try to find the places in your product that most users are converting from and ensure that you're setting context correctly the step before. One of our most successful tests on our homepage changed the primary CTA from "Start Coding" to "Sign Up." Most of the time, we were cute with copy or messaging; it hurt us. Conclusion Setting up the basics of a monetization system makes a world of difference as your company grows. The cohorts that sign up and don’t monetize are likely gone forever, so it really, really pays to set up a strong foundation. If you can get to a place with a strong checkout page, pricing structure, paywall, payment processing and churn appeasements, then you’ll be better than 90% of early-stage companies. Also published . here

Giant Mistakes and Successes: Lessons From Growing Codeacademy's Revenue From $10M To $50M

বিশাল ভুল এবং সাফল্য: কোডএকাডেমির আয় $10M থেকে $50M বৃদ্ধির থেকে শিক্ষা

Riesige Fehler und Erfolge: Lehren aus der Umsatzsteigerung der Codeacademy von 10 Millionen US-Dollar auf 50 Millionen US-Dollar

Grandes errores y éxitos: lecciones del aumento de los ingresos de Codeacademy de 10 millones de dólares a 50 millones de dólares

Erreurs et réussites géantes : leçons tirées de l'augmentation des revenus de Codeacademy de 10 millions de dollars à 50 millions de dollars

बड़ी गलतियाँ और सफलताएँ: Codeacademy का राजस्व $10M से $50M तक बढ़ने से सबक

大きな失敗と成功: Codeacademy の収益が 1,000 万ドルから 5,000 万ドルに成長したことから得た教訓

거대한 실수와 성공: Codeacademy의 수익이 1,000만 달러에서 5,000만 달러로 성장하면서 얻은 교훈

Erros e sucessos gigantes: lições do aumento da receita da Codeacademy de US$ 10 milhões para US$ 50 milhões

Dev Hatalar ve Başarılar: Codeacademy'nin Gelirini 10 Milyon Dolardan 50 Milyon Dolara Çıkarmaktan Dersler

Những sai lầm và thành công lớn: Bài học từ việc tăng doanh thu của Codeacademy từ 10 triệu USD lên 50 triệu USD

巨大的错误和成功：Codeacademy 收入从 1000 万美元增长到 5000 万美元的经验教训

At work, do you need to be a stronger weightlifter to outdo more senior guys? Not necessarily. The difference lies in learning. You will come to a point where knowledge gathering from more senior peers becomes less enriching. While reading, you might stumble upon an article describing an obscure tool or innovative technology that could solve your product’s issue. After some time, people may come to you for advice, a senior manager asks for your opinion. Fast-forward, you end up in a software architect role. Turn your learning into a routine In the beginning, the challenging thing is to get organized. Gather your knowledge sources first First, write down your small knowledge source catalog of various personal and company blogs, articles, podcasts, and newsletters. Start making the list on your own, then ask others about the top feeds they find valuable. I don’t suggest googling it, as, in the current era, there is actually too much noise and too many low-quality sources. Keep your list small, up to eight sources maximum. If you identify any source as irrelevant, out of scope, or simply not insightful enough, drop it. Quality is key. Make no compromise. Throughout the journey, some of the articles will refer to other useful blogs, thus expanding your own knowledge source. Allocate your time I recommend creating a daily learning block in your private and company calendar each morning. Mine is called . In the beginning, 30 minutes is more than enough. The more senior you get, the more of your time you can invest in learning. Read&Post In case anyone puts a conflicting meeting in your schedule, kindly ask them to reschedule unless it’s critical. Most importantly, be persistent. Preserve your intrinsic motivation. Initiate your daily routine and improve it over time. You will get used to it in a couple of weeks and others will start recognizing your extra value in a few months. It is natural to end up skipping 2–3 of your daily learning sessions each month. Get ready to spend more time than expected on it during the first week. The daily routine My learning time consists of the following: First, I go through my knowledge source list, while opening what seem to be the most -valuable posts in separate browser tabs. Then, I pick the top 1–2 articles, depending on my mood, my gut feeling, or the writer (you’ll start to recognize them over time). The top trick I recommend is to avoid the trap of todo lists treated as a pool of other potentially valuable articles worth reading in the future. You’ll never read them. Moreover, it creates an improper and negative connotation of duty in your mind, making your learning path less appealing. The only exception here is to use a ‘read later offline’ app, such as or , to tag articles you can come back to in the same day, like while you’re commuting. Pocket Raindrop Finally, I share the valuable pieces I’ve found across the company to dedicated channels ( , for example). Over time, more people will contribute too and you’ll end up learning from others. Slack Advanced tips: Grow your knowledge library I end my daily learning routine by saving my most treasured articles and sources in a separate folder or document. Reason being, it is amazing to build and have the list of wisdom and answers you trust at hand. All of a sudden, you can provide solutions in a shorter time frame, as opposed to relying on “Google help”. The invaluable side effect is, by manipulating the top quotes and links, the ideas will better persist in your mind. There are many tools out there I tried to help with tagging various sources. After several experiments, I circled back to the most simplistic approach: having a basic folder structure categorized by the domain of interest, i.e., for technical, , or leadership content. managerial Each file in the category folder represents a different topic of interest. For handling files, I prefer using markdown syntax as it helps keep the file content of links and quotes clear and understandable. The knowledge library folder is then synchronized with cloud storage (i.e., free tier) to make sure nothing gets wiped out accidentally. Eventually, create a free Github repository similar to . In terms of files management, I’m a big fan of editor due to its simplicity and great search features. No complex tooling. Google Drive this Atom I advise you to let your knowledge library grow as soon as you get comfortable with the previous steps of the daily routine, not before. Find your feed management app After you get used to spending your morning time learning, I advise trying out a feed management application. Most of them have a free tier, which already covers most of the features. My top picks are or . inoreader feedly The reason the app is worth your investment, is that it unifies your different sources of blogs, podcasts, and newsletters in one single spot, while distinguishing the read vs. unread posts, thus saving your time spent with browser tab orchestration. Manager’s conclusion In the world of software development, a manager should understand that knowledge sharing and learning are the highest background priority. Such company skill adoption is a long-term initiative, as opposed to one company OKR cycle. When we embrace the culture of intrinsic learning and knowledge sharing, we gain a competitive advantage in feature delivery, pace, and hiring. Talent attracts talent. Set the groundwork for domain-specific knowledge-sharing channels. Contribute to it yourself. Create regular knowledge-sharing sessions with an assigned owner (architect, guru, or staff-level). Make the learning and sharing responsibility explicit in your career growth scheme. Rather than ‘generously’ providing paid sources, help your people in learning habit adoption and in the sources gathering step. Create an environment where people are incentivized to post blogs. Make it explicit by allocating the time in the form of a ticket, handled by community people. Reward it. Market it. Summary Do not let your adventure die by staying in the workforce area for too long. To give you an advantage and keep one step ahead, consider reusing anything relevant in my and feed lists. Now, it’s your turn. Set the calendar morning slot. programming managerial Marian provides coaching and mentoring to various levels, helping organisations to succeed in building products. engineering managers and leaders across Stay tuned for the next post! 💪 Cover Photo by Josh Calabrese on Unsplash

Learn or Die: Turn your Learning into a Routine

Advances in learning theory recently gave a talk about a new theory of learning he is working on. The theory involves the concept of a “teacher” that provides the learning algorithm with privileged examples that can speed up learning. Vladimir Vapnik The inspiration came from the observation that humans require far more examples to learn than current machine learning algorithms. His theory is that this phenomenon is partly due to the fact that the teacher controls the examples that are given to the student, while also supplying the student with privileged information, that even though might not be available at test time, can be used to speed up training. Quite interestingly, has dealt with the same observation that humans learn much faster. His approach, called was already discussed in a previous . Joshua Tenenbaum Bayesian Program Learning post The talk can be found here https://www.youtube.com/watch?v=o6WMr1QDNCQ?embedable=true Who is Vladimir Vapnik? Vladimir Vapnik (born in 1936) is the person who discovered one of the most popular algorithms in machine learning: the . support vector machine He gained his Masters Degree in Mathematics in 1958 at Uzbek State University, Samarkand, USSR. He then worked at the Institute of Control Sciences, Moscow, from 1961 to 1990 where he became Head of the Computer Science Research Department. He then joined AT&T Bell Laboratories in New Jersey, having been appointed Professor of Computer Science and Statistics at Royal Holloway in 1995. The theory of support vector machines was influenced a lot by his other great contribution to machine learning, the theory (also known as VC theory). This was the first complete theory of computational learning, which provided theoretical bounds and guarantees for the generalization of machine learning models. Vapnik-Chervonenkis The VC theory of learning is also particularly important because it managed to provide an alternative to the Bayesian theory of machine learning which had dominated the machine learning community in the 90s. The reason is that the VC learning theory does not require the definition of any prior distributions, and thus provides a more general way to approach the problem of learning. The text on the board is a reference to a classic computer game . joke If you are interested to read more about the views of great people in data science, then I would suggest you also read the article about . Marvin Minsky Also Published Here

Vladimir Vapnik's New Learning Model

“Immigrating to Canada as a child I had this idea of what success looked like, and I pictured someone in a suit,” Saeid says. “I went into the hardest programs to get into, and ended up working in consulting and investment banking.” Although Saeid has always been fascinated with tech and design, the uncertainty in the field had him focused on business instead. “My first career was a horrible fit for me personality-wise, it tickled the intellectual curiosity to some degree…but it was hands-off,” he says. “I just had a really deep dissatisfaction with my life.” Saeid’s concept of a successful person started to morph. He wanted to be challenged intellectually and to be able to explore his creative side. Saeid wanted to build and develop applications. “I did a career 180, and started to learn coding and design,” he says. Saeid purchased books on Javascript and taught himself the foundations of coding. Soon, he’d find himself delving into novice freelancing and eventually working for Expedia as a product manager. In Seattle, Saeid found himself a little bit everywhere within Expedia, volunteering to do design work and taking any chance he could to work on the product itself. Saeid held onto a piece of advice that would resonate with individuals going through similar experiences of change; “You don’t have to make your next move perfect, but it needs to be a step in the right direction.” At the end of the day, there was still something missing, something that Expedia wasn’t providing for Saeid to fully challenge himself or pique his interest. So Saeid got on a plane straight to Vancouver and found himself joining a startup called Sokanu. Sokanu was a major building block for Saeid, spending seven years working his way up, learning and growing. From lead designer to president, Saeid became engulfed in the tech industry. In 2020, Saeid ended his time at Sokanu and founded Dimensional, a mobile app startup. It is described as a life coach in your pocket, giving users the ability to access personality tests that improve their lives and relationships by going beyond the traditional and limited personality tests that exist today. “Career testers are interesting, but it’s a very point in time solution,” he says, “It’s applicable to a very narrow slice of people.” Saeid believes there are more impactful ways to break down personalities and into something more influential in a person’s life, helping them reflect on who they are. “Let’s take a step back and reflect on who we are and tune our lives to it.” “I would wake up and have the Sunday scaries, those days were miserable, I constantly felt like a fish out of water,” he says. He recalls a moment when a friend told him he had the personality of an artist when they both lived together in Seattle during his Expedia days. For Saeid, this felt odd, as he thought he had been business-focused for so long. “I never thought of myself as an artist, and I reflected on that,” he says. “Writers, actors, people who are concerned with ideas, those are the kinds of people that I tend to get along with, that’s my tribe.” Saeid found that intrinsic connections between business and tech pulled out the creative side of him, and also helped him understand the tech world better. “This could be your advantage in life…you have training for your left brain but you’re a right-brain person.” For students, newcomers in the industry or people who just want to get their foot in the door with coding and tech, Saeid says there are a plethora of opportunities now with the new digital age. “It’s useful to take some elementary computer science courses…learning the primitives of the theoretical primitives of programming is useful and gives you the upper hand versus people who just know coding and scripting.”  Saeid says that if you truly want to build, he would recommend taking college courses to gain a stronger understanding. “You go to coding boot camp, you learn a lot of stuff about building, and you know the basic things to understand like how recursion works or certain fundamental things,” he said. “People with a more theoretical background, won’t hit a wall, they can keep going and they’ll learn faster.” Aside from the ease of accessibility to courses within computer science, is there a deeper need for change in the industry? Saeid spoke about the complexity of the industry, how it needs to be better in terms of diversity. “In Canada, tech is very Asian-centered, both South and East Asian, and while that breaks the traditional mold, it’s still very, very male [centered],” he says. “There’s some diversity, but there needs to be more female candidates applying and in engineering programs. I would say it’s improved but not incredibly.” While this does exist, Saeid had consistently found himself in physically diverse environments but emphasized the trend in cognitive diversity is usually one sided. “Engineers tend to be very analytical, very left brain,” he says. “It would be interesting to see how the field evolves but at the moment the industry has very little neurodiversity.” All in all, Saeid’s array of knowledge and ability to adapt have benefited him in the long run. For people looking to change their path or expand their learning, Saeid is a great example of how a vast amount of skills can help you grow into the individual you wish to be. “We are still in the early stages of technology and how it can change the world,” he says. “I don’t know what the future is going to look like, but there’s always going to be room for software in our lifetime.” Check out more cool computer science professionals like Saeid Fard at Thinking Cap!

UP LEARN

uplearn.co.uk

-- employees

Since n.d.

COMPANY RANKING

UP LEARN

EVERGREEN INDEX #11428

Representing this company? Click here to claim and customize this page!

Up Learn WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

UP LEARN

uplearn.co.uk

-- employees

Since n.d.

COMPANY RANKING

UP LEARN

EVERGREEN INDEX #11428

RELATED COMPANIESRANK

STORIES ABOUT UP LEARN

Representing this company? Click here to claim and customize this page!

Up Learn WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!