Gamification Boosts Software Testing—Until the Rewards Run Out

Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Editor's note: This is part 7 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Table of Links Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 5 Discussion 6 Threats to validity 7 Conclusions, Acknowledgements and References 7 Conclusions This work presents a long gamification experience that was designed and conducted to motivate the students to create effective test suites, and makes a comparison with a non-gamified experience. The results show that the gamification benefits the improvement of both student engagement and performance. We have also observed a slight reduction of the engagement in some students toward the last stage of the experience, when they perceived that no more rewards were going to be received by keeping on working on the gamification experience. The statistical analysis indicates that the differences are significant. In addition, the study confirms how the design of the gamification strategies is crucial for engaging students. The results obtained show that the rewards in this gamification experience acted as powerful extrinsic motivators that kept the students motivated until they perceived they were not going to get more rewards for their work. Arguably, because of the particular gamification design of this experience, the link between gamification and intrinsic motivation could not be proved. Nevertheless, the results did prove that gamification succeeds in keeping extrinsic motivation and improving performance for long periods of time. While designing a gamification experience, the professors should distribute the motivating stimulus throughout the whole experience, so that the engagement lasts until the end. As a part of our future work, we will readjust the design of the gamification experience to include new rewards that increase student engagement in the last seminar exercises, so that we can overcome the challenge of keeping the engagement from decreasing slightly toward the last stage of the experience. In addition, we will study the effects of gamification in our software testing course during several academic years as part of a longitudinal study. Acknowledgements This work was supported by projects PID2019-105455GB-C32 and PID2019-105455GB-C33 funded by MCIN/AEI/10.13039/501100011033 (Spain). References Alhammad, M. M., & Moreno, A. M. (2018). Gamification in software engineering education: A systematic mapping. Journal of Systems and Software, 141, 131150. https://doi.org/10.1016/j.jss.2018.03.065 Andrade, S. A., De Oliveira Neves, V., & Delamaro, M. E. (2019). Software testing education: Dreams and challenges when bringing academia and industry closer together. XXXIII Brazilian Symposium on Software Engineering, SBES 2019, 47 56. https://doi.org/10.1145/3350768.3353903 Bell, J., Sheth, S., & Kaiser, G. (2011). Secret ninja testing with HALO software engineering. 4th International Workshop on Social Software Engineering, SSE 2011. https://doi.org/10.1145/2024645.2024657 Bodnar, C. A., Anastasio, D., Enszer, J. A., & Burkey, D. D. (2016). Engineers at Play: Games as Teaching Tools for Undergraduate Engineering Students. Journal of Engineering Education, 105(1), 147 200. https://doi.org/10.1002/jee.20106 Bourque, P., & Fairley, R. E. (2014). SWEBOK Guide V3.0. In IEEE Computer Society. Buckley, I. A., & Clarke, P. J. (2018). An approach to teaching software testing supported by two different online content delivery methods. 16th LACCEI International Multi-Conference for Engineering, Education and Technology, 2018-July. https://doi.org/10.18687/LACCEI2018.1.1.377 Cacciotto, F., Fulcini, T., Coppola, R., & Ardito, L. (2021). A metric framework for the gamification of web and mobile GUI testing. IEEE 14th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2021, 126129. https://doi.org/10.1109/ICSTW52544.2021.00032 Calderón, A., Boubeta-Puig, J., & Ruiz, M. (2018). MEdit4CEP-Gam: A model-driven approach for user-friendly gamification design, monitoring and code generation in CEP-based systems. Information and Software Technology, 95, 238 264. https://doi.org/10.1016/j.infsof.2017.11.009 Capgemini. (2021). World Quality Report 2020-2021. Clarke, P. J., Davis, D. L., Buckley, I. A., Potvin, G., Thirunarayanan, M., & Jones, E. L. (2019). An approach to integrating learning and engagement strategies (LESS) into CS class activities. ASEE Annual Conference and Exposition. https://doi.org/10.18260/1-2--32057 Clarke, P. J., Davis, D. L., Buckley, I. A., Potvin, G., Thirunarayanan, M., & Jones, E. L. (2022). Combining Learning and Engagement Strategies in a Software Testing Learning Environment. ACM Transactions on Computing Education, 22(2), 1 25. https://doi.org/10.1145/3469131 Clarke, P. J., Davis, D. L., Chang-Lau, R., & King, T. M. (2017). Impact of Using Tools in an Undergraduate Software Testing Course Supported by WReSTT. ACM Transactions on Computing Education, 17(4). https://doi.org/10.1145/3068324 Clarke, P. J., Thirunarayanan, M., Allala, S. C., Sotomayor, J. P., & Ross, M. S. (2020). Experiences of integrating learning and engagement strategies (LESs) into software engineering courses. ASEE Annual Conference and Exposition, 2020-June. https://doi.org/10.18260/1-2--34630 Clegg, B. S., Rojas, J. M., & Fraser, G. (2017). Teaching software testing concepts using a mutation testing game. 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering and Education Track, ICSE-SEET 2017, 33 36. https://doi.org/10.1109/ICSESEET.2017.1 Costa, I. E. F., & Oliveira, S. R. B. (2019). A systematic strategy to teaching of exploratory testing using gamification. 14th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2019, 307 314. https://doi.org/10.5220/0007711603070314 Costa, I. E. F., & Oliveira, S. R. B. (2020). The use of gamification to support the teaching-learning of software exploratory testing: An experience report based on the application of a framework. IEEE Frontiers in Education Conference, FIE 2020, 2020-October, 19. https://doi.org/10.1109/FIE44824.2020.9273943 Dal Sasso, T., Mocci, A., Lanza, M., & Mastrodicasa, E. (2017). How to gamify software engineering. 24th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2017, 261 271. https://doi.org/10.1109/SANER.2017.7884627 Deak, A., Stålhane, T., & Sindre, G. (2016). Challenges and strategies for motivating software testing personnel. Information and Software Technology, 73, 115. https://doi.org/10.1016/j.infsof.2016.01.002 Deterding, S., O’Hara, K., Sicat. M., Dixon., & Nacker, L (2021). Gamification: Using game design elements in non-gaming contexts. Extended Abstracts on Human Factors in Computing Systems, CHI EA 2011, 2425 2428. https://doi.org/10.1145/1979742.1979575 Dichev, C., & Dicheva, D. (2017). Gamifying education: what is known, what is believed and what remains uncertain: a critical review. International Journal of Educational Technology in Higher Education, 14. https://doi.org/10.1186/s41239-017-0042-5 Elbaum, S., Person, S., Dokulil, J., & Jorde, M. (2007). Bug hunt: Making early software testing lessons engaging and affordable. 29th International Conference on Software Engineering, ICSE 2007, 688 697. https://doi.org/10.1109/ICSE.2007.23 Elgrably, I. S., & Oliveira, S. R. B. (2018). Gamification and evaluation of the use the agile tests in software quality subjects: The application of case studies. 13th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2018, 2018-March, 416423. https://doi.org/10.5220/0006800304160423 Elgrably, I. S., & Oliveira, S. R. B. (2022). Using flipped classroom to promote active learning and engagement in a Software Testing subject remotely during the COVID-19 pandemic. IEEE Frontiers in Education Conference, FIE 2022, 16. https://doi.org/10.1109/FIE56618.2022.9962379 Fraser, G. (2017). Gamification of Software Testing. 12th IEEE/ACM International Workshop on Automation of Software Testing, AST 2017, 2 7. https://doi.org/10.1109/AST.2017.20 Fraser, G., Gambi, A., Kreis, M., & Rojas, J. M. (2019). Gamifying a Software Testing Course with Code Defenders. 50th ACM Technical Symposium on Computer Science Education, SIGCSE 2019, 571 577. https://doi.org/10.1145/3287324.3287471 Fraser, G., Gambi, A., & Rojas, J. M. (2020). Teaching Software Testing with the Code Defenders Testing Game:Experiences and Improvements. 13th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2020, 461464. https://doi.org/10.1109/ICSTW50294.2020.00082 Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59109. https://doi.org/10.3102/00346543074001059 Fu, Y., & Clarke, P. J. (2016). Gamification-based cyber-enabled learning environment of software testing. ASEE Annual Conference and Exposition, 2016-June. https://doi.org/10.18260/p.27000 Garaccione, G., Fulcini, T., & Torchiano, M. (2022). GERRY: a gamified browser tool for GUI testing. 1st International Workshop on Gamification of Software Development, Verification and Validation, Gamify 2022, 2 9. https://doi.org/10.1145/3548771.3561408 Garcia, I., Pacheco, C., Méndez, F., & Calvo-Manzano, J. A. (2020). The effects of game-based learning in the acquisition of “soft skills“on undergraduate software engineering courses: A systematic literature review. Computer Applications in Engineering Education, 28(5), 13271354. https://doi.org/10.1002/cae.22304 Garousi, V., Rainer, A., Lauvås, P., & Arcuri, A. (2020). Software-testing education: A systematic literature mapping. Journal of Systems and Software, 165. https://doi.org/10.1016/J.JSS.2020.110570 Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? - A literature review of empirical studies on gamification. 47th Hawaii International Conference on System Sciences, 30253034. https://doi.org/10.1109/HICSS.2014.377 International Software Testing Qualification Board (ISTQB). (2020). ISTQB Standard glossary of terms used in Software Testing, version 3.5. https://glossary.istqb.org/ Isomöttönen, V., & Lappalainen, V. (2012). CS1 with games and an emphasis on TDD and unit testing: Piling a trend upon a trend. ACM Inroads, 3(3), 62 68. https://doi.org/10.1145/2339055.2339073 Jesus, G. M. de, Ferrari, F. C., Paschoal, L. N., Souza, S. D. R. S. de, Porto, D. D. P., & Durelli, V. H. S. (2020). Is It Worth Using Gamification on Software Testing Education? An Extended Experience Report in the Context of Undergraduate Students. Journal of Software Engineering Research and Development, 8, 6:1-6:19. https://doi.org/10.5753/jserd.2020.738 Jesus, G. M. de, Ferrari, F. C., Porto, D. de P., & Fabbri, S. C. P. F. (2018). Gamification in software testing: A characterization study. III Brazilian Symposium on Systematic and Automated Software Testing, SAST 2018, 39 48. https://doi.org/10.1145/3266003.3266007 Khaleel, F. L., Ashaari, N. S., Wook, T. S. M. T. W. T., & Ismail, A. (2016). The Architecture of Dynamic Gamification Elements Based Learning Content. Journal of Convergence Information Technology, 11(3), 164 177. Khandelwal, S., Sripada, S. K., & Raghu Reddy, Y. (2017). Impact of gamification on code review process - An experimental study. 10th Innovations in Software Engineering Conference, ISEC 2017, 122 126. https://doi.org/10.1145/3021460.3021474 Kosa, M., Yilmaz, M., O'Connor, R. V., & Clarke, P. M. (2016). Software engineering education and games: A systematic literature review. Journal of Universal Computer Science, 22(12), 1558– 1574. https://doi.org/10.3217/jucs-022-12-1558 Krasner, H. (2021). The Cost of Poor Software Quality in the US: A 2020 Report. Lőrincz, B., Iudean, B., & Vescan, A. (2021). Experience report on teaching testing through gamification. 3rd International Workshop on Education through Advanced Software Engineering and Artificial Intelligence, EASEAI 2021, 15-22. https://doi.org/10.1145/3472673.3473960 Marabesi, M., & Silveira, I. F. (2019). Towards a gamified tool to improve unit test teaching. 14th Latin American Conference on Learning Technologies, LACLO 2019, 1218. https://doi.org/10.1109/LACLO49268.2019.00013 Micari, M., & Pazos, P. (2012). Connecting to the Professor: Impact of the Student Faculty Relationship in a Highly Challenging Course. College Teaching, 60(2), 4147. https://doi.org/10.1080/87567555.2011.627576 Milosz, M., & Milosz, E. (2020). Gamification in engineering education - A preliminary literature review. IEEE Global Engineering Education Conference, EDUCON 2020, 2020-April. https://doi.org/10.1109/EDUCON45650.2020.9125108 Myers, G. J., Badgett, T., & Sandler, C. (2012). The art of software testing (3rd Edition). Wiley. Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y. Le, & Harman, M. (2019). Mutation Testing Advances: An Analysis and Survey. Advances in Computers, 112, 275378. https://doi.org/10.1016/bs.adcom.2018.03.015 Pedreira, O., García, F., Brisaboa, N., & Piattini, M. (2015). Gamification in software engineering - A systematic mapping. Information and Software Technology, 57, 157168. https://doi.org/10.1016/j.infsof.2014.08.007 Peixoto, M., & Silva, C. (2017). A gamification requirements catalog for educational software: Results from a systematic literature review and a survey with experts. ACM Symposium on Applied Computing, Part F128005, 1108 1113. https://doi.org/10.1145/3019612.3019752 Rojas, J. M., & Fraser, G. (2016). Code Defenders: A Mutation Testing Game. 9th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016, 162167. https://doi.org/10.1109/ICSTW.2016.43 Ruiperez-Valiente, J. A., Gomez, M. J., Martinez, P. A., & Kim, Y. J. (2021). Ideating and Developing a Visualization Dashboard to Support Teachers Using Educational Games in the Classroom. IEEE Access, 9, 83467 83481. https://doi.org/10.1109/ACCESS.2021.3086703 Sardi, L., Idri, A., & Fernández-Alemán, J. L. (2017). A systematic review of gamification in e-Health. Journal of Biomedical Informatics, 71, 31 48. https://doi.org/10.1016/j.jbi.2017.05.011 Sherif, E., Liu, A., Nguyen, B., Lerner, S., & Griswold, W. G. (2020). Gamification to Aid the Learning of Test Coverage Concepts. 32nd IEEE International Conference on Software Engineering Education and Training, CSEE&T 2020, 320324. https://doi.org/10.1109/CSEET49119.2020.9206224 Sheth, S., Bell, J., & Kaiser, G. (2015). A Gameful approach to teaching software design and software testing. In Computer Games and Software Engineering (pp. 91113). https://doi.org/10.1201/b18453 Sheth, S., Bell, J., & Kaiser, G. (2013). A competitive-collaborative approach for introducing software engineering in a CS2 class. 26th International Conference on Software Engineering Education and Training, CSEE&T 2013, 41 50. https://doi.org/10.1109/CSEET.2013.6595235 Silvis-Cividjian, N. (2021). Awesome Bug Manifesto: Teaching an Engaging and Inspiring Course on Software Testing (Position Paper). 3rd International Workshop on Software Engineering Education for the Next Generation, SEENG 2021, 1620. https://doi.org/10.1109/SEENG53126.2021.00010 Souza, M. R. d. A., Veado, L., Moreira, R. T., Figueiredo, E., & Costa, H. (2018). A systematic mapping study on game-related methods for software engineering education. Information and Software Technology, 95, 201 218. https://doi.org/10.1016/j.infsof.2017.09.014 Subhash, S., & Cudney, E. A. (2018). Gamified learning in higher education: A systematic review of the literature. Computers in Human Behavior, 87, 192206. https://doi.org/10.1016/j.chb.2018.05.028 Trinidad, M., Calderon, A., & Ruiz, M. (2021). GoRace: A Multi-Context and Narrative-Based Gamification Suite to Overcome Gamification Technological Challenges. IEEE Access, 9, 6588265905. https://doi.org/10.1109/ACCESS.2021.3076291 Valle, P. H. D., Toda, A. M., Barbosa, E. F., & Maldonado, J. C. (2017). Educational games: A contribution to software testing education. IEEE Frontiers in Education Conference, FIE 2017, 2017-October, 1 8. https://doi.org/10.1109/FIE.2017.8190470 Vos, T. E. J., Fraser, G., Martinez-Ortiz, I., Prada, R., Silva, A. R., & Prasetya, I. S. W. B. (2020). Tutorial on a Gamification Toolset for Improving Engagement of Studentsin Software Engineering Courses. In M. Daun, E. Hochmuller, S. Krusche, B. Brugge, & B. Tenbergen (Eds.), 32nd IEEE Conference on Software Engineering Education and Training, CSEE&T 2020 (pp. 291 293). IEEE. Werbach, K., & Hunter, D. (2012). For the Win: How Game Thinking Can Revolutionize Your Business. Wharton School Press. Zivkovic, T., & Zivkovic, M. (2021). Survey of Learning Environments for Software Testing Education. 7th Conference on the Engineering of Computer Based Systems, ECBS 2021, 19. https://doi.org/10.1145/3459960.3459971 This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Authors: Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Editor's note: This is part 7 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 7 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 7 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 7 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Table of Links Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 5 Discussion 6 Threats to validity 7 Conclusions, Acknowledgements and References Abstract and 1 Introduction Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 3.2 Participants 3.2 Participants 3.3 Materials 3.3 Materials 3.4 Procedure 3.4 Procedure 3.5 Metrics 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 4.2 RQ2: Student performance 5 Discussion 5 Discussion 6 Threats to validity 6 Threats to validity 7 Conclusions, Acknowledgements and References 7 Conclusions, Acknowledgements and References 7 Conclusions This work presents a long gamification experience that was designed and conducted to motivate the students to create effective test suites, and makes a comparison with a non-gamified experience. The results show that the gamification benefits the improvement of both student engagement and performance. We have also observed a slight reduction of the engagement in some students toward the last stage of the experience, when they perceived that no more rewards were going to be received by keeping on working on the gamification experience. The statistical analysis indicates that the differences are significant. In addition, the study confirms how the design of the gamification strategies is crucial for engaging students. The results obtained show that the rewards in this gamification experience acted as powerful extrinsic motivators that kept the students motivated until they perceived they were not going to get more rewards for their work. Arguably, because of the particular gamification design of this experience, the link between gamification and intrinsic motivation could not be proved. Nevertheless, the results did prove that gamification succeeds in keeping extrinsic motivation and improving performance for long periods of time. While designing a gamification experience, the professors should distribute the motivating stimulus throughout the whole experience, so that the engagement lasts until the end. As a part of our future work, we will readjust the design of the gamification experience to include new rewards that increase student engagement in the last seminar exercises, so that we can overcome the challenge of keeping the engagement from decreasing slightly toward the last stage of the experience. In addition, we will study the effects of gamification in our software testing course during several academic years as part of a longitudinal study. Acknowledgements Acknowledgements This work was supported by projects PID2019-105455GB-C32 and PID2019-105455GB-C33 funded by MCIN/AEI/10.13039/501100011033 (Spain). References Alhammad, M. M., & Moreno, A. M. (2018). Gamification in software engineering education: A systematic mapping. Journal of Systems and Software, 141, 131150. https://doi.org/10.1016/j.jss.2018.03.065 Andrade, S. A., De Oliveira Neves, V., & Delamaro, M. E. (2019). Software testing education: Dreams and challenges when bringing academia and industry closer together. XXXIII Brazilian Symposium on Software Engineering, SBES 2019, 47 56. https://doi.org/10.1145/3350768.3353903 Bell, J., Sheth, S., & Kaiser, G. (2011). Secret ninja testing with HALO software engineering. 4th International Workshop on Social Software Engineering, SSE 2011. https://doi.org/10.1145/2024645.2024657 Bodnar, C. A., Anastasio, D., Enszer, J. A., & Burkey, D. D. (2016). Engineers at Play: Games as Teaching Tools for Undergraduate Engineering Students. Journal of Engineering Education, 105(1), 147 200. https://doi.org/10.1002/jee.20106 Bourque, P., & Fairley, R. E. (2014). SWEBOK Guide V3.0. In IEEE Computer Society. Buckley, I. A., & Clarke, P. J. (2018). An approach to teaching software testing supported by two different online content delivery methods. 16th LACCEI International Multi-Conference for Engineering, Education and Technology, 2018-July. https://doi.org/10.18687/LACCEI2018.1.1.377 Cacciotto, F., Fulcini, T., Coppola, R., & Ardito, L. (2021). A metric framework for the gamification of web and mobile GUI testing. IEEE 14th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2021, 126129. https://doi.org/10.1109/ICSTW52544.2021.00032 Calderón, A., Boubeta-Puig, J., & Ruiz, M. (2018). MEdit4CEP-Gam: A model-driven approach for user-friendly gamification design, monitoring and code generation in CEP-based systems. Information and Software Technology, 95, 238 264. https://doi.org/10.1016/j.infsof.2017.11.009 Capgemini. (2021). World Quality Report 2020-2021. Clarke, P. J., Davis, D. L., Buckley, I. A., Potvin, G., Thirunarayanan, M., & Jones, E. L. (2019). An approach to integrating learning and engagement strategies (LESS) into CS class activities. ASEE Annual Conference and Exposition. https://doi.org/10.18260/1-2--32057 Clarke, P. J., Davis, D. L., Buckley, I. A., Potvin, G., Thirunarayanan, M., & Jones, E. L. (2022). Combining Learning and Engagement Strategies in a Software Testing Learning Environment. ACM Transactions on Computing Education, 22(2), 1 25. https://doi.org/10.1145/3469131 Clarke, P. J., Davis, D. L., Chang-Lau, R., & King, T. M. (2017). Impact of Using Tools in an Undergraduate Software Testing Course Supported by WReSTT. ACM Transactions on Computing Education, 17(4). https://doi.org/10.1145/3068324 Clarke, P. J., Thirunarayanan, M., Allala, S. C., Sotomayor, J. P., & Ross, M. S. (2020). Experiences of integrating learning and engagement strategies (LESs) into software engineering courses. ASEE Annual Conference and Exposition, 2020-June. https://doi.org/10.18260/1-2--34630 Clegg, B. S., Rojas, J. M., & Fraser, G. (2017). Teaching software testing concepts using a mutation testing game. 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering and Education Track, ICSE-SEET 2017, 33 36. https://doi.org/10.1109/ICSESEET.2017.1 Costa, I. E. F., & Oliveira, S. R. B. (2019). A systematic strategy to teaching of exploratory testing using gamification. 14th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2019, 307 314. https://doi.org/10.5220/0007711603070314 Costa, I. E. F., & Oliveira, S. R. B. (2020). The use of gamification to support the teaching-learning of software exploratory testing: An experience report based on the application of a framework. IEEE Frontiers in Education Conference, FIE 2020, 2020-October, 19. https://doi.org/10.1109/FIE44824.2020.9273943 Dal Sasso, T., Mocci, A., Lanza, M., & Mastrodicasa, E. (2017). How to gamify software engineering. 24th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2017, 261 271. https://doi.org/10.1109/SANER.2017.7884627 Deak, A., Stålhane, T., & Sindre, G. (2016). Challenges and strategies for motivating software testing personnel. Information and Software Technology, 73, 115. https://doi.org/10.1016/j.infsof.2016.01.002 Deterding, S., O’Hara, K., Sicat. M., Dixon., & Nacker, L (2021). Gamification: Using game design elements in non-gaming contexts. Extended Abstracts on Human Factors in Computing Systems, CHI EA 2011, 2425 2428. https://doi.org/10.1145/1979742.1979575 Dichev, C., & Dicheva, D. (2017). Gamifying education: what is known, what is believed and what remains uncertain: a critical review. International Journal of Educational Technology in Higher Education, 14. https://doi.org/10.1186/s41239-017-0042-5 Elbaum, S., Person, S., Dokulil, J., & Jorde, M. (2007). Bug hunt: Making early software testing lessons engaging and affordable. 29th International Conference on Software Engineering, ICSE 2007, 688 697. https://doi.org/10.1109/ICSE.2007.23 Elgrably, I. S., & Oliveira, S. R. B. (2018). Gamification and evaluation of the use the agile tests in software quality subjects: The application of case studies. 13th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2018, 2018-March, 416423. https://doi.org/10.5220/0006800304160423 Elgrably, I. S., & Oliveira, S. R. B. (2022). Using flipped classroom to promote active learning and engagement in a Software Testing subject remotely during the COVID-19 pandemic. IEEE Frontiers in Education Conference, FIE 2022, 16. https://doi.org/10.1109/FIE56618.2022.9962379 Fraser, G. (2017). Gamification of Software Testing. 12th IEEE/ACM International Workshop on Automation of Software Testing, AST 2017, 2 7. https://doi.org/10.1109/AST.2017.20 Fraser, G., Gambi, A., Kreis, M., & Rojas, J. M. (2019). Gamifying a Software Testing Course with Code Defenders. 50th ACM Technical Symposium on Computer Science Education, SIGCSE 2019, 571 577. https://doi.org/10.1145/3287324.3287471 Fraser, G., Gambi, A., & Rojas, J. M. (2020). Teaching Software Testing with the Code Defenders Testing Game:Experiences and Improvements. 13th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2020, 461464. https://doi.org/10.1109/ICSTW50294.2020.00082 Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59109. https://doi.org/10.3102/00346543074001059 Fu, Y., & Clarke, P. J. (2016). Gamification-based cyber-enabled learning environment of software testing. ASEE Annual Conference and Exposition, 2016-June. https://doi.org/10.18260/p.27000 Garaccione, G., Fulcini, T., & Torchiano, M. (2022). GERRY: a gamified browser tool for GUI testing. 1st International Workshop on Gamification of Software Development, Verification and Validation, Gamify 2022, 2 9. https://doi.org/10.1145/3548771.3561408 Garcia, I., Pacheco, C., Méndez, F., & Calvo-Manzano, J. A. (2020). The effects of game-based learning in the acquisition of “soft skills“on undergraduate software engineering courses: A systematic literature review. Computer Applications in Engineering Education, 28(5), 13271354. https://doi.org/10.1002/cae.22304 Garousi, V., Rainer, A., Lauvås, P., & Arcuri, A. (2020). Software-testing education: A systematic literature mapping. Journal of Systems and Software, 165. https://doi.org/10.1016/J.JSS.2020.110570 Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? - A literature review of empirical studies on gamification. 47th Hawaii International Conference on System Sciences, 30253034. https://doi.org/10.1109/HICSS.2014.377 International Software Testing Qualification Board (ISTQB). (2020). ISTQB Standard glossary of terms used in Software Testing, version 3.5. https://glossary.istqb.org/ Isomöttönen, V., & Lappalainen, V. (2012). CS1 with games and an emphasis on TDD and unit testing: Piling a trend upon a trend. ACM Inroads, 3(3), 62 68. https://doi.org/10.1145/2339055.2339073 Jesus, G. M. de, Ferrari, F. C., Paschoal, L. N., Souza, S. D. R. S. de, Porto, D. D. P., & Durelli, V. H. S. (2020). Is It Worth Using Gamification on Software Testing Education? An Extended Experience Report in the Context of Undergraduate Students. Journal of Software Engineering Research and Development, 8, 6:1-6:19. https://doi.org/10.5753/jserd.2020.738 Jesus, G. M. de, Ferrari, F. C., Porto, D. de P., & Fabbri, S. C. P. F. (2018). Gamification in software testing: A characterization study. III Brazilian Symposium on Systematic and Automated Software Testing, SAST 2018, 39 48. https://doi.org/10.1145/3266003.3266007 Khaleel, F. L., Ashaari, N. S., Wook, T. S. M. T. W. T., & Ismail, A. (2016). The Architecture of Dynamic Gamification Elements Based Learning Content. Journal of Convergence Information Technology, 11(3), 164 177. Khandelwal, S., Sripada, S. K., & Raghu Reddy, Y. (2017). Impact of gamification on code review process - An experimental study. 10th Innovations in Software Engineering Conference, ISEC 2017, 122 126. https://doi.org/10.1145/3021460.3021474 Kosa, M., Yilmaz, M., O'Connor, R. V., & Clarke, P. M. (2016). Software engineering education and games: A systematic literature review. Journal of Universal Computer Science, 22(12), 1558– 1574. https://doi.org/10.3217/jucs-022-12-1558 https://doi.org/10.3217/jucs-022-12-1558 Krasner, H. (2021). The Cost of Poor Software Quality in the US: A 2020 Report. Lőrincz, B., Iudean, B., & Vescan, A. (2021). Experience report on teaching testing through gamification. 3rd International Workshop on Education through Advanced Software Engineering and Artificial Intelligence, EASEAI 2021, 15-22. https://doi.org/10.1145/3472673.3473960 https://doi.org/10.1145/3472673.3473960 Marabesi, M., & Silveira, I. F. (2019). Towards a gamified tool to improve unit test teaching. 14th Latin American Conference on Learning Technologies, LACLO 2019, 1218. https://doi.org/10.1109/LACLO49268.2019.00013 Micari, M., & Pazos, P. (2012). Connecting to the Professor: Impact of the Student Faculty Relationship in a Highly Challenging Course. College Teaching, 60(2), 4147. https://doi.org/10.1080/87567555.2011.627576 Milosz, M., & Milosz, E. (2020). Gamification in engineering education - A preliminary literature review. IEEE Global Engineering Education Conference, EDUCON 2020, 2020-April. https://doi.org/10.1109/EDUCON45650.2020.9125108 Myers, G. J., Badgett, T., & Sandler, C. (2012). The art of software testing (3rd Edition). Wiley. Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y. Le, & Harman, M. (2019). Mutation Testing Advances: An Analysis and Survey. Advances in Computers, 112, 275378. https://doi.org/10.1016/bs.adcom.2018.03.015 Pedreira, O., García, F., Brisaboa, N., & Piattini, M. (2015). Gamification in software engineering - A systematic mapping. Information and Software Technology, 57, 157168. https://doi.org/10.1016/j.infsof.2014.08.007 Peixoto, M., & Silva, C. (2017). A gamification requirements catalog for educational software: Results from a systematic literature review and a survey with experts. ACM Symposium on Applied Computing, Part F128005, 1108 1113. https://doi.org/10.1145/3019612.3019752 Rojas, J. M., & Fraser, G. (2016). Code Defenders: A Mutation Testing Game. 9th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016, 162167. https://doi.org/10.1109/ICSTW.2016.43 Ruiperez-Valiente, J. A., Gomez, M. J., Martinez, P. A., & Kim, Y. J. (2021). Ideating and Developing a Visualization Dashboard to Support Teachers Using Educational Games in the Classroom. IEEE Access, 9, 83467 83481. https://doi.org/10.1109/ACCESS.2021.3086703 Sardi, L., Idri, A., & Fernández-Alemán, J. L. (2017). A systematic review of gamification in e-Health. Journal of Biomedical Informatics, 71, 31 48. https://doi.org/10.1016/j.jbi.2017.05.011 Sherif, E., Liu, A., Nguyen, B., Lerner, S., & Griswold, W. G. (2020). Gamification to Aid the Learning of Test Coverage Concepts. 32nd IEEE International Conference on Software Engineering Education and Training, CSEE&T 2020, 320324. https://doi.org/10.1109/CSEET49119.2020.9206224 Sheth, S., Bell, J., & Kaiser, G. (2015). A Gameful approach to teaching software design and software testing. In Computer Games and Software Engineering (pp. 91113). https://doi.org/10.1201/b18453 Sheth, S., Bell, J., & Kaiser, G. (2013). A competitive-collaborative approach for introducing software engineering in a CS2 class. 26th International Conference on Software Engineering Education and Training, CSEE&T 2013, 41 50. https://doi.org/10.1109/CSEET.2013.6595235 Silvis-Cividjian, N. (2021). Awesome Bug Manifesto: Teaching an Engaging and Inspiring Course on Software Testing (Position Paper). 3rd International Workshop on Software Engineering Education for the Next Generation, SEENG 2021, 1620. https://doi.org/10.1109/SEENG53126.2021.00010 Souza, M. R. d. A., Veado, L., Moreira, R. T., Figueiredo, E., & Costa, H. (2018). A systematic mapping study on game-related methods for software engineering education. Information and Software Technology, 95, 201 218. https://doi.org/10.1016/j.infsof.2017.09.014 Subhash, S., & Cudney, E. A. (2018). Gamified learning in higher education: A systematic review of the literature. Computers in Human Behavior, 87, 192206. https://doi.org/10.1016/j.chb.2018.05.028 Trinidad, M., Calderon, A., & Ruiz, M. (2021). GoRace: A Multi-Context and Narrative-Based Gamification Suite to Overcome Gamification Technological Challenges. IEEE Access, 9, 6588265905. https://doi.org/10.1109/ACCESS.2021.3076291 Valle, P. H. D., Toda, A. M., Barbosa, E. F., & Maldonado, J. C. (2017). Educational games: A contribution to software testing education. IEEE Frontiers in Education Conference, FIE 2017, 2017-October, 1 8. https://doi.org/10.1109/FIE.2017.8190470 Vos, T. E. J., Fraser, G., Martinez-Ortiz, I., Prada, R., Silva, A. R., & Prasetya, I. S. W. B. (2020). Tutorial on a Gamification Toolset for Improving Engagement of Studentsin Software Engineering Courses. In M. Daun, E. Hochmuller, S. Krusche, B. Brugge, & B. Tenbergen (Eds.), 32nd IEEE Conference on Software Engineering Education and Training, CSEE&T 2020 (pp. 291 293). IEEE. Werbach, K., & Hunter, D. (2012). For the Win: How Game Thinking Can Revolutionize Your Business. Wharton School Press. Zivkovic, T., & Zivkovic, M. (2021). Survey of Learning Environments for Software Testing Education. 7th Conference on the Engineering of Computer Based Systems, ECBS 2021, 19. https://doi.org/10.1145/3459960.3459971 This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv