Effective Use of Snowballing in Gamification Research

Authors: (1) Clauvin Almeida, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (2) Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (3) Anderson Uchoa, Federal University of Ceara (UFC), Itapaje, Brazil; (4) Bruno Feijo, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil. Table of Links Abstract and 1 Introduction 2. Background and Related Work and 2.1. Gamification 2.2. Game Design Elements and 2.3. Gamification Effects 2.4. Related Work on Gamification Negative Effects 3. Systematic Mapping and 3.1. The Research Questions 3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria 3.4. Applying the Search Strategy 3.5. Data Extraction 4. Systematic Mapping Results 5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements 5.1. Context and Participant Characterization 5.2. Focus Group Design 5.3. The Developers’ Perception on The Negative Effects 5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects 5.5. Participant Feedback 6. Limitations 7. Concluding Remarks 7.1. Future Research Directions Acknowledgements and References 3.4. Applying the Search Strategy 3.4.1. Search strategy application reported in our previous study We first applied the search string on Scopus on July 28th, 2020, searching within the title, abstract, and keywords. It returned 180 documents, upon which we applied the exclusion criteria through three filtering phases, as described in Table 6. After this initial filtering a set of 64 papers remained. Thereafter, still as part of our conference paper efforts [14], we conducted backward and forward snowballing using these 64 papers as seed set, both on August 18th, 2020. The papers retrieved from backward snowballing and from forward snowballing using Scopus citation information were merged with the seed set, resulting in 2338 unique entries. Additionally, considering that Mourao et al. [15] suggest using Google Scholar for forward snowballing, besides doing it using Scopus citation information, we also conducted forward snowballing using citation information from Google Scholar (on September 4th, 2020). The forward snowballing through Google Scholar found 738 additional unique entries. Hence, we ended up with 3076 unique entries (including the seed set 64). We applied our inclusion and exclusion criteria to the title, abstract, and keywords of the 3012 papers retrieved through snowballing, as shown in Table 7. After the title, abstract, and keyword filtering, we conducted full-text-based filtering for the remaining 140 papers. The result of this full-text-based filtering is shown in Table 8, resulting in a set of 68 included papers. Out of those, 32 were found by the initial Scopus search, 18 by forward snowballing, 15 by backward snowballing, and 3 were retrieved by both forward and backward snowballing. These numbers also help to illustrate how snowballing can be complementary to database searches. We emphasize that we conducted the full-text-based assessment only after snowballing on purpose, as we thought that applying snowballing on some additional closely related papers would not be detrimental. Nevertheless, this decision indeed increased our snowballing effort. Finally, still as part of our previous effort reported in [14], to complement our search strategy, we compared our set of 68 included papers against the 17 papers included by [27]. While our set of 68 papers to be included comprised 29 papers ranging from 2012 to 2016, only seven of them were also included by [27]; i.e., their search strategy did not retrieve 22 papers reporting negative effects of gamification in education/learning software that were retrieved by our search strategy. On the other hand, our search strategy missed nine papers included in their mapping (the remaining one was retrieved but eliminated from our mapping for not being related to “digital” GDEs – EC7). As a result of this comparison, to present a mapping including all papers that we were aware of, we manually included the papers found by [27] that were missed by our search strategy, ending up with a final set of 77 included papers for our initial publication [14]. 3.4.2. Extending the search efforts A natural extension strategy would be covering the gap of papers published until the end of 2020, and conducting additional forward and backward snowballing iterations. As forward snowballing based on previously included papers is an effective strategy for updating systematic literature studies [51], the update until the end of 2020 could be accomplished by applying forward snowballing to the set of 68 papers identified initially as part of our strategy. Similarly, we could conduct additional backward snowballing iterations (i.e., on the papers retrieved through the first forward and backward iterations). We checked upon the feasibility of applying this extension strategy with reasonable effort. We noticed that, a second snowballing iteration on the 68 papers, keeping our search temporally upper limited to the end of 2020, would involve analyzing a total of 5632 additional entries (1275 from backward snowballing and 4357 from forward snowballing, with a small overlap between both searches). Unfortunately, despite our best efforts from October 22, 2021 to January 18, 2022, this amount of entries proved to be unfeasible to handle as part of this extension. We understood that complementing snowballing iterations until saturation would involve analyzing several thousands of papers and characterize enough effort for a completely new paper. All the data from the unfinished second snowballing iteration is available in our Zenodo repository (www.doi.org/10.5281/zenodo.6279062). Therefore, we decided on a different strategy to assure feasibility within a reasonable manuscript extension effort. With the intent to address the gap involving papers from the 2nd semester of 2020 with reasonable effort, our strategy involved: (i) re-executing our original search string on Scopus, limiting results until the end of 2020; (ii) filtering these papers; and (iii) applying forward and backward snowballing on the additional included papers. Thus, in January 18, 2022 we re-executed our original search string on Scopus, searching within the title, abstract, and keywords. The search retrieved 266 documents. Out of these, 59 were excluded for being from 2021 or beyond. We identified that 176 of these papers were also identified in our similar search conducted in July 28, 2020 (we cannot explain why it did not retrieve all 180 previously returned papers, as we executed the exact same search string). The title, abstract, and keyword based filtering of the remaining ones is shown in Table 9. The exclusion criteria applied was the same as before, with the difference that we did one filtering phase instead of three, covering all the exclusion criteria. After full-text-based filtering (as shown in Table 10) a set of 4 papers remained. Then, we conducted backward and forward snowballing using these 4 papers as a seed set, on January 29, 2022. Google Scholar’s backward snowballing retrieved 192 unique entries, Scopus forward snowballing retrieved 11 and Google Scholar forward snowballing retrieved 2. In total, backward and forward snowballing retrieving 205 entries. We applied the title, abstract, and keyword filtering on these 205 entries, resulting in 44 papers, as shown in Table 11. Thereafter, we conducted full- textbased filtering for the remaining 44 papers, as shown in Table 12, resulting in a set of 7 additional papers. Including the 4 papers obtained from the search string filtering, the search strategy extension identified 11 additional papers. Extending the set of papers retrieved through our search from 68 to 79. At all, considering the original search effort and this extension, out of the 79 papers, 36 papers were found by the initial Scopus searches, 18 by forward snowballing, 22 by backward snowballing, and 3 were retrieved by both forward and backward snowballing. Again, these numbers help to illustrate how snowballing can be complementary to database searches. When comparing our new set of 79 included papers against the 17 papers included by [27], while our set of 79 papers to be included comprised 34 papers ranging from 2012 to 2016, only eight of them were also included by [27] i.e., their search strategy did not retrieve 26 papers reporting negative effects of gamification in education/learning software that were retrieved by our search strategy. On the other hand, our search strategy missed eight papers included in their mapping (the remaining one was retrieved but eliminated from our mapping for not being related to “digital” GDEs – EC7). Indeed, after the second search that found the eleven additional papers, we found out that of the nine papers missed by the original search, one appeared as part of the backward snowballing of the second search. Furthermore, we noticed that four additional missed papers were part of the set of papers to be analyzed for the second snowballing iteration. While, considering the effort, as previously justified, we could not apply snowballing until saturation, we believe that the remaining four papers found by [27] and not by us would probably be retrieved as part of subsequent snowballing iterations. On the other hand, the effort of snowballing may be unfeasible within popular topics of research, such as gamification, leading to several thousands of papers to be analyzed. Aiming at present a mapping including all papers that we were aware of, we manually included the eight papers found by [27] that were missed by our search strategy, ending up with a set of 87 included papers. In summary, the scope of our search strategy (analyzing more than 3500 papers) and the added value (we extended the previously mapped evidence from 17 to 87 papers) provides an unbiased and meaningful overview on the adverse effects of gamification in educational software. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Clauvin Almeida, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (2) Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (3) Anderson Uchoa, Federal University of Ceara (UFC), Itapaje, Brazil; (4) Bruno Feijo, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil. Authors: Authors: (1) Clauvin Almeida, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (2) Marcos Kalinowski, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil; (3) Anderson Uchoa, Federal University of Ceara (UFC), Itapaje, Brazil; (4) Bruno Feijo, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Background and Related Work and 2.1. Gamification 2. Background and Related Work and 2.1. Gamification 2.2. Game Design Elements and 2.3. Gamification Effects 2.2. Game Design Elements and 2.3. Gamification Effects 2.4. Related Work on Gamification Negative Effects 2.4. Related Work on Gamification Negative Effects 3. Systematic Mapping and 3.1. The Research Questions 3. Systematic Mapping and 3.1. The Research Questions 3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria 3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria 3.4. Applying the Search Strategy 3.4. Applying the Search Strategy 3.5. Data Extraction 3.5. Data Extraction 4. Systematic Mapping Results 4. Systematic Mapping Results 5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements 5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements 5.1. Context and Participant Characterization 5.1. Context and Participant Characterization 5.2. Focus Group Design 5.2. Focus Group Design 5.3. The Developers’ Perception on The Negative Effects 5.3. The Developers’ Perception on The Negative Effects 5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects 5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects 5.5. Participant Feedback 5.5. Participant Feedback 6. Limitations 6. Limitations 7. Concluding Remarks 7. Concluding Remarks 7.1. Future Research Directions 7.1. Future Research Directions Acknowledgements and References Acknowledgements and References 3.4. Applying the Search Strategy 3.4. Applying the Search Strategy 3.4.1. Search strategy application reported in our previous study 3.4.1. Search strategy application reported in our previous study 3.4.1. Search strategy application reported in our previous study We first applied the search string on Scopus on July 28th, 2020, searching within the title, abstract, and keywords. It returned 180 documents, upon which we applied the exclusion criteria through three filtering phases, as described in Table 6. After this initial filtering a set of 64 papers remained. Thereafter, still as part of our conference paper efforts [14], we conducted backward and forward snowballing using these 64 papers as seed set, both on August 18th, 2020. The papers retrieved from backward snowballing and from forward snowballing using Scopus citation information were merged with the seed set, resulting in 2338 unique entries. Additionally, considering that Mourao et al. [15] suggest using Google Scholar for forward snowballing, besides doing it using Scopus citation information, we also conducted forward snowballing using citation information from Google Scholar (on September 4th, 2020). The forward snowballing through Google Scholar found 738 additional unique entries. Hence, we ended up with 3076 unique entries (including the seed set 64). We applied our inclusion and exclusion criteria to the title, abstract, and keywords of the 3012 papers retrieved through snowballing, as shown in Table 7. After the title, abstract, and keyword filtering, we conducted full-text-based filtering for the remaining 140 papers. The result of this full-text-based filtering is shown in Table 8, resulting in a set of 68 included papers. Out of those, 32 were found by the initial Scopus search, 18 by forward snowballing, 15 by backward snowballing, and 3 were retrieved by both forward and backward snowballing. These numbers also help to illustrate how snowballing can be complementary to database searches. We emphasize that we conducted the full-text-based assessment only after snowballing on purpose, as we thought that applying snowballing on some additional closely related papers would not be detrimental. Nevertheless, this decision indeed increased our snowballing effort. Finally, still as part of our previous effort reported in [14], to complement our search strategy, we compared our set of 68 included papers against the 17 papers included by [27]. While our set of 68 papers to be included comprised 29 papers ranging from 2012 to 2016, only seven of them were also included by [27]; i.e., their search strategy did not retrieve 22 papers reporting negative effects of gamification in education/learning software that were retrieved by our search strategy. On the other hand, our search strategy missed nine papers included in their mapping (the remaining one was retrieved but eliminated from our mapping for not being related to “digital” GDEs – EC7). As a result of this comparison, to present a mapping including all papers that we were aware of, we manually included the papers found by [27] that were missed by our search strategy, ending up with a final set of 77 included papers for our initial publication [14]. 3.4.2. Extending the search efforts 3.4.2. Extending the search efforts 3.4.2. Extending the search efforts A natural extension strategy would be covering the gap of papers published until the end of 2020, and conducting additional forward and backward snowballing iterations. As forward snowballing based on previously included papers is an effective strategy for updating systematic literature studies [51], the update until the end of 2020 could be accomplished by applying forward snowballing to the set of 68 papers identified initially as part of our strategy. Similarly, we could conduct additional backward snowballing iterations (i.e., on the papers retrieved through the first forward and backward iterations). We checked upon the feasibility of applying this extension strategy with reasonable effort. We noticed that, a second snowballing iteration on the 68 papers, keeping our search temporally upper limited to the end of 2020, would involve analyzing a total of 5632 additional entries (1275 from backward snowballing and 4357 from forward snowballing, with a small overlap between both searches). Unfortunately, despite our best efforts from October 22, 2021 to January 18, 2022, this amount of entries proved to be unfeasible to handle as part of this extension. We understood that complementing snowballing iterations until saturation would involve analyzing several thousands of papers and characterize enough effort for a completely new paper. All the data from the unfinished second snowballing iteration is available in our Zenodo repository (www.doi.org/10.5281/zenodo.6279062). Therefore, we decided on a different strategy to assure feasibility within a reasonable manuscript extension effort. With the intent to address the gap involving papers from the 2nd semester of 2020 with reasonable effort, our strategy involved: (i) re-executing our original search string on Scopus, limiting results until the end of 2020; (ii) filtering these papers; and (iii) applying forward and backward snowballing on the additional included papers. Thus, in January 18, 2022 we re-executed our original search string on Scopus, searching within the title, abstract, and keywords. The search retrieved 266 documents. Out of these, 59 were excluded for being from 2021 or beyond. We identified that 176 of these papers were also identified in our similar search conducted in July 28, 2020 (we cannot explain why it did not retrieve all 180 previously returned papers, as we executed the exact same search string). The title, abstract, and keyword based filtering of the remaining ones is shown in Table 9. The exclusion criteria applied was the same as before, with the difference that we did one filtering phase instead of three, covering all the exclusion criteria. After full-text-based filtering (as shown in Table 10) a set of 4 papers remained. Then, we conducted backward and forward snowballing using these 4 papers as a seed set, on January 29, 2022. Google Scholar’s backward snowballing retrieved 192 unique entries, Scopus forward snowballing retrieved 11 and Google Scholar forward snowballing retrieved 2. In total, backward and forward snowballing retrieving 205 entries. We applied the title, abstract, and keyword filtering on these 205 entries, resulting in 44 papers, as shown in Table 11. Thereafter, we conducted full- textbased filtering for the remaining 44 papers, as shown in Table 12, resulting in a set of 7 additional papers. Including the 4 papers obtained from the search string filtering, the search strategy extension identified 11 additional papers. Extending the set of papers retrieved through our search from 68 to 79. At all, considering the original search effort and this extension, out of the 79 papers, 36 papers were found by the initial Scopus searches, 18 by forward snowballing, 22 by backward snowballing, and 3 were retrieved by both forward and backward snowballing. Again, these numbers help to illustrate how snowballing can be complementary to database searches. When comparing our new set of 79 included papers against the 17 papers included by [27], while our set of 79 papers to be included comprised 34 papers ranging from 2012 to 2016, only eight of them were also included by [27] i.e., their search strategy did not retrieve 26 papers reporting negative effects of gamification in education/learning software that were retrieved by our search strategy. On the other hand, our search strategy missed eight papers included in their mapping (the remaining one was retrieved but eliminated from our mapping for not being related to “digital” GDEs – EC7). Indeed, after the second search that found the eleven additional papers, we found out that of the nine papers missed by the original search, one appeared as part of the backward snowballing of the second search. Furthermore, we noticed that four additional missed papers were part of the set of papers to be analyzed for the second snowballing iteration. While, considering the effort, as previously justified, we could not apply snowballing until saturation, we believe that the remaining four papers found by [27] and not by us would probably be retrieved as part of subsequent snowballing iterations. On the other hand, the effort of snowballing may be unfeasible within popular topics of research, such as gamification, leading to several thousands of papers to be analyzed. Aiming at present a mapping including all papers that we were aware of, we manually included the eight papers found by [27] that were missed by our search strategy, ending up with a set of 87 included papers. In summary, the scope of our search strategy (analyzing more than 3500 papers) and the added value (we extended the previously mapped evidence from 17 to 87 papers) provides an unbiased and meaningful overview on the adverse effects of gamification in educational software. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv