Researchers Think Gamification Holds the Answer to Software Testing Woes

Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Editor's note: This is part 1 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Table of Links Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 5 Discussion 6 Threats to validity 7 Conclusions, Acknowledgements and References Abstract Software testing is an essential knowledge area required by industry for software engineers. However, software engineering students often consider testing less appealing than designing or coding. Consequently, it is difficult to engage students to create effective tests. To encourage students, we explored the use of gamification and investigated whether this technique can help to improve the engagement and performance of software testing students. We conducted a controlled experiment to compare the engagement and performance of two groups of students that took an undergraduate software testing course in different academic years. The experimental group is formed by 135 students from the gamified course whereas the control group is formed by 100 students from the non-gamified course. The data collected were statistically analyzed to answer the research questions of this study. The results show that the students that participated in the gamification experience were more engaged and achieved a better performance. As an additional finding, the analysis of the results reveals that a key aspect to succeed is the gamification experience design. It is important to distribute the motivating stimulus provided by the gamification throughout the whole experience to engage students until the end. Given these results, we plan to readjust the gamification experience design to increase student engagement in the last stage of the experience, as well as to conduct a longitudinal study to evaluate the effects of gamification. Keywords Gamification, software testing, engagement, performance, test effectiveness, empirical study 1 Introduction In the Information Technology era, software is present in almost every aspect of our lives. It is crucial that software products work correctly and as intended, because their malfunction could produce financial loss to both final users and software companies, loss of prestige to software companies, exposure of private data, or even tragic consequences in human life. In order to prevent these negative effects, it is essential to test properly and thoroughly the software products with the aim of detecting and solving their defects (also called faults or bugs) and failures before releasing them to final users. For that purpose, industry requires well-prepared testers and education has a responsibility for that issue. However, there is a gap between industry and education regarding the testing knowledge needed in the industry and what is taught in higher education institutions. On the one hand, software testing is not taught with enough depth in the software engineering education (Fraser et al., 2019; Jesus et al., 2020) and it is often embedded in other software engineering courses instead of being a dedicated course (Silvis-Cividjian, 2021). On the other hand, it is difficult to engage students as they tend to find software testing a boring and destructive task, and they prefer the design and implementation tasks (Fraser et al., 2020). As a result, students do not acquire the practice that the industry requires and they prefer to work in other software engineering areas rather than in software testing; as stated in (Silvis-Cividjian, 2021), only up to 20% of students consider a career in software testing. To deal with the problem of motivating students, we have explored the use of gamification in a software testing undergraduate course. Gamification has been adopted in the last decade as a means to engage students in different educational contexts (Bodnar et al., 2016; Dichev & Dicheva, 2017; Milosz & Milosz, 2020; Subhash & Cudney, 2018; Vos et al., 2020), and has given promising results in software engineering and software testing education (Alhammad & Moreno, 2018; Fraser, 2017; Garousi et al., 2020; Jesus et al., 2018; Kosa et al., 2016; Pedreira et al., 2015; Souza et al., 2018). In this paper, we describe and evaluate the gamification experience carried out in seminar classes of a software testing course, where students had to create effective test cases for several faulty programs in order to detect their defects. We hypothesized that gamification would positively impact on student engagement and performance. The students were involved in a narrative-based gamification experience where they took part in a legendary Olympic race for immortality, supported by a gamification tool called GoRace. The effectiveness of their test cases determined their progress in the race. This effectiveness is provided by a tool called SQLTest, which is integrated with GoRace. To test our hypotheses, we defined a set of metrics to measure the student engagement and performance and statistically compared the results achieved by the experimental and the control groups in order to answer the following research questions: RQ1. Is the engagement of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To address this research question, we defined a set of metrics related to the student's participation in the academic activities. RQ2. Is the performance of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To answer this research question, we defined a set of metrics related to the effectiveness of the test cases created. The main contributions of this work are as follows: The design of a gamification experience for an undergraduate software testing course. This experience was carried out along a complete academic semester (15 weeks). An analysis of the impact of gamification on improving the student engagement and their performance in the creation of effective test cases. The results indicate that gamification has a positive impact on both the student engagement and performance; in addition, they also reveal that it is crucial to maintain the stimulus during the whole experience to ensure success. The rest of this paper is organized as follows: Section 2 provides a background on gamification and software testing education. Section 3 presents the course involved in the gamification experience, along with the participants and the materials used in the experience. This section also describes the procedure followed in the experience and the metrics defined. Section 4 presents the results of the statistical analysis carried out to test the hypotheses that allow us to answer the research questions. These results are discussed in Section 5. Section 6 lists the threats to the validity of the findings discussed. Finally, we draw our conclusions and describe the implications of our findings in Section 7. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Authors: Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Editor's note: This is part 1 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 1 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 1 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 1 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Table of Links Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 5 Discussion 6 Threats to validity 7 Conclusions, Acknowledgements and References Abstract and 1 Introduction Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 3.2 Participants 3.2 Participants 3.3 Materials 3.3 Materials 3.4 Procedure 3.4 Procedure 3.5 Metrics 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 4.2 RQ2: Student performance 5 Discussion 5 Discussion 6 Threats to validity 6 Threats to validity 7 Conclusions, Acknowledgements and References 7 Conclusions, Acknowledgements and References Abstract Software testing is an essential knowledge area required by industry for software engineers. However, software engineering students often consider testing less appealing than designing or coding. Consequently, it is difficult to engage students to create effective tests. To encourage students, we explored the use of gamification and investigated whether this technique can help to improve the engagement and performance of software testing students. We conducted a controlled experiment to compare the engagement and performance of two groups of students that took an undergraduate software testing course in different academic years. The experimental group is formed by 135 students from the gamified course whereas the control group is formed by 100 students from the non-gamified course. The data collected were statistically analyzed to answer the research questions of this study. The results show that the students that participated in the gamification experience were more engaged and achieved a better performance. As an additional finding, the analysis of the results reveals that a key aspect to succeed is the gamification experience design. It is important to distribute the motivating stimulus provided by the gamification throughout the whole experience to engage students until the end. Given these results, we plan to readjust the gamification experience design to increase student engagement in the last stage of the experience, as well as to conduct a longitudinal study to evaluate the effects of gamification. Keywords Keywords Gamification, software testing, engagement, performance, test effectiveness, empirical study 1 Introduction In the Information Technology era, software is present in almost every aspect of our lives. It is crucial that software products work correctly and as intended, because their malfunction could produce financial loss to both final users and software companies, loss of prestige to software companies, exposure of private data, or even tragic consequences in human life. In order to prevent these negative effects, it is essential to test properly and thoroughly the software products with the aim of detecting and solving their defects (also called faults or bugs) and failures before releasing them to final users. For that purpose, industry requires well-prepared testers and education has a responsibility for that issue. However, there is a gap between industry and education regarding the testing knowledge needed in the industry and what is taught in higher education institutions. On the one hand, software testing is not taught with enough depth in the software engineering education (Fraser et al., 2019; Jesus et al., 2020) and it is often embedded in other software engineering courses instead of being a dedicated course (Silvis-Cividjian, 2021). On the other hand, it is difficult to engage students as they tend to find software testing a boring and destructive task, and they prefer the design and implementation tasks (Fraser et al., 2020). As a result, students do not acquire the practice that the industry requires and they prefer to work in other software engineering areas rather than in software testing; as stated in (Silvis-Cividjian, 2021), only up to 20% of students consider a career in software testing. To deal with the problem of motivating students, we have explored the use of gamification in a software testing undergraduate course. Gamification has been adopted in the last decade as a means to engage students in different educational contexts (Bodnar et al., 2016; Dichev & Dicheva, 2017; Milosz & Milosz, 2020; Subhash & Cudney, 2018; Vos et al., 2020), and has given promising results in software engineering and software testing education (Alhammad & Moreno, 2018; Fraser, 2017; Garousi et al., 2020; Jesus et al., 2018; Kosa et al., 2016; Pedreira et al., 2015; Souza et al., 2018). In this paper, we describe and evaluate the gamification experience carried out in seminar classes of a software testing course, where students had to create effective test cases for several faulty programs in order to detect their defects. We hypothesized that gamification would positively impact on student engagement and performance. The students were involved in a narrative-based gamification experience where they took part in a legendary Olympic race for immortality, supported by a gamification tool called GoRace. The effectiveness of their test cases determined their progress in the race. This effectiveness is provided by a tool called SQLTest, which is integrated with GoRace. To test our hypotheses, we defined a set of metrics to measure the student engagement and performance and statistically compared the results achieved by the experimental and the control groups in order to answer the following research questions: RQ1. Is the engagement of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To address this research question, we defined a set of metrics related to the student's participation in the academic activities. RQ2. Is the performance of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To answer this research question, we defined a set of metrics related to the effectiveness of the test cases created. RQ1. Is the engagement of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To address this research question, we defined a set of metrics related to the student's participation in the academic activities. RQ1. Is the engagement of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To address this research question, we defined a set of metrics related to the student's participation in the academic activities. RQ2. Is the performance of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To answer this research question, we defined a set of metrics related to the effectiveness of the test cases created. RQ2. Is the performance of the software testing students who carry out gamified activities higher than the ones who carry them out in a non-gamified environment? To answer this research question, we defined a set of metrics related to the effectiveness of the test cases created. The main contributions of this work are as follows: The design of a gamification experience for an undergraduate software testing course. This experience was carried out along a complete academic semester (15 weeks). An analysis of the impact of gamification on improving the student engagement and their performance in the creation of effective test cases. The results indicate that gamification has a positive impact on both the student engagement and performance; in addition, they also reveal that it is crucial to maintain the stimulus during the whole experience to ensure success. The design of a gamification experience for an undergraduate software testing course. This experience was carried out along a complete academic semester (15 weeks). The design of a gamification experience for an undergraduate software testing course. This experience was carried out along a complete academic semester (15 weeks). An analysis of the impact of gamification on improving the student engagement and their performance in the creation of effective test cases. The results indicate that gamification has a positive impact on both the student engagement and performance; in addition, they also reveal that it is crucial to maintain the stimulus during the whole experience to ensure success. An analysis of the impact of gamification on improving the student engagement and their performance in the creation of effective test cases. The results indicate that gamification has a positive impact on both the student engagement and performance; in addition, they also reveal that it is crucial to maintain the stimulus during the whole experience to ensure success. The rest of this paper is organized as follows: Section 2 provides a background on gamification and software testing education. Section 3 presents the course involved in the gamification experience, along with the participants and the materials used in the experience. This section also describes the procedure followed in the experience and the metrics defined. Section 4 presents the results of the statistical analysis carried out to test the hypotheses that allow us to answer the research questions. These results are discussed in Section 5. Section 6 lists the threats to the validity of the findings discussed. Finally, we draw our conclusions and describe the implications of our findings in Section 7. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv