Mapping and Addressing Gaps in Negative Effects of GDEs

Authors: (1) Clauvin Almeida, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (2) Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (3) Anderson Uchoa, Federal University of Ceara (UFC), Itapaje, Brazil; (4) Bruno Feijo, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil. Table of Links Abstract and 1 Introduction 2. Background and Related Work and 2.1. Gamification 2.2. Game Design Elements and 2.3. Gamification Effects 2.4. Related Work on Gamification Negative Effects 3. Systematic Mapping and 3.1. The Research Questions 3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria 3.4. Applying the Search Strategy 3.5. Data Extraction 4. Systematic Mapping Results 5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements 5.1. Context and Participant Characterization 5.2. Focus Group Design 5.3. The Developers’ Perception on The Negative Effects 5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects 5.5. Participant Feedback 6. Limitations 7. Concluding Remarks 7.1. Future Research Directions Acknowledgements and References 4. Systematic Mapping Results Figure 1 shows the distribution of the 87 included papers throughout the publication years. Results for each of our research questions based on the extracted data follow. 4.1. RQ1 - What game design elements cause which negative effects in the field of education/learning? Overall, the papers reported 94 different GDEs, 69 different negative effects caused to the user, and ten different negative effects caused to the person maintaining the software or in the role of a teacher. We decided to conservatively ground the answer to this question on associations reported by more than one paper instead of correlation and causation, strengthening, in the process, our confidence in the results. We adopted this approach, considering the vast amount of different GDEs and effects, plus the fact that many of the papers found had GDEs grouped and used together instead of individually, plus the fact that future research may show that some negative effects may be caused by a poor implementation of a GDE, instead of the GDE itself. The complete extracted data, allowing different analyses, can be found in our open science repository. 4.1.1. RQ1.A – What game design elements caused negative effects in the field of education/learning? Figure 2 summarizes the GDEs mentioned by at least two papers and the number of papers that referred to each of them as causing negative effects. The list of papers referring to each element can be identified in the online repository. It is possible to observe that most of the reported negative effects were associated with the use of badges, leaderboards, competitions and points. This makes sense given that these are GDEs commonly used in gamification, which may be related to creating competitive environments. It is also noteworthy that there were several (77) other GDEs, which had only one paper each indicating negative effects. Further analysis is required to answer whether this can be explained by the lack of negative effects caused by these elements or the lack of investigations involving them. 4.1.2. RQ1.B - What negative effects of game design elements were found affecting those interacting with the software as users or being in the role of a student? Figure 3 shows the negative effects caused to the user mentioned by at least two papers and the number of times that papers referenced those negative effects. It is possible to observe that the most cited negative effects concern the lack of effect, worsened performance, demotivation, lack of motivation, and lack of understanding. The ethical issues of gaming the system and cheating were also recurrent reported effects, usually motivated by creating competitive reward environments (which stimulated users to break the rules to beat the competition) and/or systems with failures that enable users to easily score by cheating. Discouragement and dislike of competition were also noticeable. Typically, if the student does not like competition and is losing in terms of grades, the visible gap between himself and those ahead will not result in improvement, but in losing hope [53]. Dislike of gamification and alienation or confusion for non-gamers appeared as well. Gamification is not a generic solution that works for any person, nor something that should be considered known by everyone. The ”alienation or confusion” cases made exactly that mistake and then had to deal with users that misunderstood how the system worked. For instance, a point-based system replacing a grading system, where the students did not understand how many points they needed to get a passing grade, and because of that lost many opportunities to increase their grades until it was too late [54]. Again, several (75) other negative effects caused to the user were mentioned only once. Hence, the most common negative effect was that using the gamified software resulted in no difference when compared to not using the gamified software. Someone may argue that the negative effects characterized as being the “lack of” something are not negative, given that nothing bad effectively happened. However, for each of such neutral results to happen, gamification elements were designed and implemented, requiring human effort, that, while helping to show what does not work in terms of educational technology, effectively made no difference to the learners. Another effect that needs explanation is the “Novelty effect”, which is a negative effect, in the sense that potential positive effects may be temporary. i.e., as soon as the user’s interest goes away, the positive effects will not apply anymore, and if they were not present for enough time, they may not be enough in terms of cost/benefit. 4.1.3. RQ1.C – Which negative effects of game design elements were found affecting those maintaining the software or being in the role of a teacher? In this question, we grouped both teachers and those who maintain the software working as both deal with the part of the gamified software that the students usually don’t touch, and end up having to fix/circumvent problems that appear. Table 13 shows the negative effects caused to those maintaining the software working or being in the position of a teacher, which were mentioned more than once and the number of times those negative effects were mentioned within the analyzed papers. It is possible to observe that the most common negative effects concern technical challenges and extra required effort or resources. Extra human effort and resources needed typically appear as a negative effect when the gamified software imply having to create additional content and taking care of additional tasks on top of the everyday tasks related to education. Finally, engineering problems typically appeared when learning management software did not cover what the designers wanted them to do, leading to implementation workarounds and potentially lower quality.[1] Seven other negative effects were cited only once. 4.1.4. Wrapping up RQ1 - What game design elements cause which negative effects in the field of education/learning? To complete the answer to RQ1, we mapped the GDEs against the related negative effects. The bubble plot in Figure 4 shows the GDE and negative effect pairings that appeared more than once in our systematic mapping. This mapping can help raise gamification designers’ awareness of potential undesired negative effects of GDEs on education/learning software. It is possible to observe, for instance, that the use of Badges may have no effect [55][56][57][58][59][60], end up being irrelevant [55][61][62][63][64], or even result in worsened performance [59][65][66][67][68], amidst other reported negative effects, such as time pressure, bugs. Similar interpretations can be made for the remaining GDEs. We provide a table containing the specific references related to the bubble plot of Figure 4 in Table 14. It is noteworthy that the primary studies included in our mapping vary in context and empirical strategy. Hence, a more in depth analyses for each GDE would involve carefully analyzing its related research papers, which are not easy to aggregate as part of a broadly scoped secondary study. 4.2. RQ2 - In what fields of education/learning were the negative effects of game design elements found? The fields where negative effects of GDEs were reported more than once are shown in Figure 5. Besides the listed ones, there were 29 other fields reported that were cited only once. It can be observed that the negative effects were reported in several different areas. Given the closeness between games, gamification, and digital technology, computer science being the most covered subject was expected. 4.3. RQ3 – What types of empirical studies were conducted to assess the negative effects? To answer this question, we used Wohlin’s classification [105], which divides empirical strategies into surveys, case studies, and controlled experiments. As shown in Table 15, most of the research was reported as concerning case studies or controlled experiments, complemented by surveys. The positive aspect is that all papers reported applied at least one empirical strategy, which is expected in papers concerning the observation of effects. Overall, the mapping study identified empirical studies revealing negative effects of GDEs. It is possible to observe a concentration of research on a small subset of negative effects of educational software GDEs (cf. Figure 4 and Table 14). There is still limited research outside of that subset pointing to research gaps that the community could address. Hereafter we enrich the discussion on the mapping study results through a focus group, debating the mapped negative effects of the GDEs with developers of gamified software. This paper is available on arxiv under CC BY 4.0 DEED license. [1] One of the ”engineering problems with the LMS” was also counted as ”extra general human effort needed”, as it ended causing extra effort as well Authors: (1) Clauvin Almeida, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (2) Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (3) Anderson Uchoa, Federal University of Ceara (UFC), Itapaje, Brazil; (4) Bruno Feijo, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil. Authors: Authors: (1) Clauvin Almeida, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (2) Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (3) Anderson Uchoa, Federal University of Ceara (UFC), Itapaje, Brazil; (4) Bruno Feijo, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Background and Related Work and 2.1. Gamification 2. Background and Related Work and 2.1. Gamification 2.2. Game Design Elements and 2.3. Gamification Effects 2.2. Game Design Elements and 2.3. Gamification Effects 2.4. Related Work on Gamification Negative Effects 2.4. Related Work on Gamification Negative Effects 3. Systematic Mapping and 3.1. The Research Questions 3. Systematic Mapping and 3.1. The Research Questions 3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria 3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria 3.4. Applying the Search Strategy 3.4. Applying the Search Strategy 3.5. Data Extraction 3.5. Data Extraction 4. Systematic Mapping Results 4. Systematic Mapping Results 5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements 5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements 5.1. Context and Participant Characterization 5.1. Context and Participant Characterization 5.2. Focus Group Design 5.2. Focus Group Design 5.3. The Developers’ Perception on The Negative Effects 5.3. The Developers’ Perception on The Negative Effects 5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects 5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects 5.5. Participant Feedback 5.5. Participant Feedback 6. Limitations 6. Limitations 7. Concluding Remarks 7. Concluding Remarks 7.1. Future Research Directions 7.1. Future Research Directions Acknowledgements and References Acknowledgements and References 4. Systematic Mapping Results Figure 1 shows the distribution of the 87 included papers throughout the publication years. Results for each of our research questions based on the extracted data follow. 4.1. RQ1 - What game design elements cause which negative effects in the field of education/learning? Overall, the papers reported 94 different GDEs, 69 different negative effects caused to the user, and ten different negative effects caused to the person maintaining the software or in the role of a teacher. We decided to conservatively ground the answer to this question on associations reported by more than one paper instead of correlation and causation, strengthening, in the process, our confidence in the results. We adopted this approach, considering the vast amount of different GDEs and effects, plus the fact that many of the papers found had GDEs grouped and used together instead of individually, plus the fact that future research may show that some negative effects may be caused by a poor implementation of a GDE, instead of the GDE itself. The complete extracted data, allowing different analyses, can be found in our open science repository. 4.1.1. RQ1.A – What game design elements caused negative effects in the field of education/learning? 4.1.1. RQ1.A – What game design elements caused negative effects in the field of education/learning? 4.1.1. RQ1.A – What game design elements caused negative effects in the field of education/learning? Figure 2 summarizes the GDEs mentioned by at least two papers and the number of papers that referred to each of them as causing negative effects. The list of papers referring to each element can be identified in the online repository. It is possible to observe that most of the reported negative effects were associated with the use of badges, leaderboards, competitions and points. This makes sense given that these are GDEs commonly used in gamification, which may be related to creating competitive environments. It is also noteworthy that there were several (77) other GDEs, which had only one paper each indicating negative effects. Further analysis is required to answer whether this can be explained by the lack of negative effects caused by these elements or the lack of investigations involving them. 4.1.2. RQ1.B - What negative effects of game design elements were found affecting those interacting with the software as users or being in the role of a student? 4.1.2. RQ1.B - What negative effects of game design elements were found affecting those interacting with the software as users or being in the role of a student? 4.1.2. RQ1.B - What negative effects of game design elements were found affecting those interacting with the software as users or being in the role of a student? Figure 3 shows the negative effects caused to the user mentioned by at least two papers and the number of times that papers referenced those negative effects. It is possible to observe that the most cited negative effects concern the lack of effect, worsened performance, demotivation, lack of motivation, and lack of understanding. The ethical issues of gaming the system and cheating were also recurrent reported effects, usually motivated by creating competitive reward environments (which stimulated users to break the rules to beat the competition) and/or systems with failures that enable users to easily score by cheating. Discouragement and dislike of competition were also noticeable. Typically, if the student does not like competition and is losing in terms of grades, the visible gap between himself and those ahead will not result in improvement, but in losing hope [53]. Dislike of gamification and alienation or confusion for non-gamers appeared as well. Gamification is not a generic solution that works for any person, nor something that should be considered known by everyone. The ”alienation or confusion” cases made exactly that mistake and then had to deal with users that misunderstood how the system worked. For instance, a point-based system replacing a grading system, where the students did not understand how many points they needed to get a passing grade, and because of that lost many opportunities to increase their grades until it was too late [54]. Again, several (75) other negative effects caused to the user were mentioned only once. Hence, the most common negative effect was that using the gamified software resulted in no difference when compared to not using the gamified software. Someone may argue that the negative effects characterized as being the “lack of” something are not negative, given that nothing bad effectively happened. However, for each of such neutral results to happen, gamification elements were designed and implemented, requiring human effort, that, while helping to show what does not work in terms of educational technology, effectively made no difference to the learners. Another effect that needs explanation is the “Novelty effect”, which is a negative effect, in the sense that potential positive effects may be temporary. i.e., as soon as the user’s interest goes away, the positive effects will not apply anymore, and if they were not present for enough time, they may not be enough in terms of cost/benefit. 4.1.3. RQ1.C – Which negative effects of game design elements were found affecting those maintaining the software or being in the role of a teacher? 4.1.3. RQ1.C – Which negative effects of game design elements were found affecting those maintaining the software or being in the role of a teacher? 4.1.3. RQ1.C – Which negative effects of game design elements were found affecting those maintaining the software or being in the role of a teacher? In this question, we grouped both teachers and those who maintain the software working as both deal with the part of the gamified software that the students usually don’t touch, and end up having to fix/circumvent problems that appear. Table 13 shows the negative effects caused to those maintaining the software working or being in the position of a teacher, which were mentioned more than once and the number of times those negative effects were mentioned within the analyzed papers. It is possible to observe that the most common negative effects concern technical challenges and extra required effort or resources. Extra human effort and resources needed typically appear as a negative effect when the gamified software imply having to create additional content and taking care of additional tasks on top of the everyday tasks related to education. Finally, engineering problems typically appeared when learning management software did not cover what the designers wanted them to do, leading to implementation workarounds and potentially lower quality.[1] Seven other negative effects were cited only once. 4.1.4. Wrapping up RQ1 - What game design elements cause which negative effects in the field of education/learning? 4.1.4. Wrapping up RQ1 - What game design elements cause which negative effects in the field of education/learning? 4.1.4. Wrapping up RQ1 - What game design elements cause which negative effects in the field of education/learning? To complete the answer to RQ1, we mapped the GDEs against the related negative effects. The bubble plot in Figure 4 shows the GDE and negative effect pairings that appeared more than once in our systematic mapping. This mapping can help raise gamification designers’ awareness of potential undesired negative effects of GDEs on education/learning software. It is possible to observe, for instance, that the use of Badges may have no effect [55][56][57][58][59][60], end up being irrelevant [55][61][62][63][64], or even result in worsened performance [59][65][66][67][68], amidst other reported negative effects, such as time pressure, bugs. Similar interpretations can be made for the remaining GDEs. We provide a table containing the specific references related to the bubble plot of Figure 4 in Table 14. It is noteworthy that the primary studies included in our mapping vary in context and empirical strategy. Hence, a more in depth analyses for each GDE would involve carefully analyzing its related research papers, which are not easy to aggregate as part of a broadly scoped secondary study. 4.2. RQ2 - In what fields of education/learning were the negative effects of game design elements found? The fields where negative effects of GDEs were reported more than once are shown in Figure 5. Besides the listed ones, there were 29 other fields reported that were cited only once. It can be observed that the negative effects were reported in several different areas. Given the closeness between games, gamification, and digital technology, computer science being the most covered subject was expected. 4.3. RQ3 – What types of empirical studies were conducted to assess the negative effects? To answer this question, we used Wohlin’s classification [105], which divides empirical strategies into surveys, case studies, and controlled experiments. As shown in Table 15, most of the research was reported as concerning case studies or controlled experiments, complemented by surveys. The positive aspect is that all papers reported applied at least one empirical strategy, which is expected in papers concerning the observation of effects. Overall, the mapping study identified empirical studies revealing negative effects of GDEs. It is possible to observe a concentration of research on a small subset of negative effects of educational software GDEs (cf. Figure 4 and Table 14). There is still limited research outside of that subset pointing to research gaps that the community could address. Hereafter we enrich the discussion on the mapping study results through a focus group, debating the mapped negative effects of the GDEs with developers of gamified software. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [1] One of the ”engineering problems with the LMS” was also counted as ”extra general human effort needed”, as it ended causing extra effort as well