How Far Can Gamification Take Software Testing?

Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Editor's note: This is part 5 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Table of Links Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 5 Discussion 6 Threats to validity 7 Conclusions, Acknowledgements and References 5 Discussion The main challenge we addressed in the present study was to measure and evaluate the impact of gamification on student engagement and performance. To deal with this challenge, we defined the metrics presented in Section 3.5 and analyzed the results obtained in the controlled experiment that was carried out. The results show that, overall, gamification contributes to the improvement of the student engagement and performance. These findings are in line with those of most works that reported practical gamification experiences in software testing education. (Buckley & Clarke, 2018; Lorincz et al., 2021; Sherif et al., 2020) compared the results achieved by gamified and non-gamified groups; (Lorincz et al., 2021; al., 2021; Sherif et al., 2020) reported that the engagement was higher in the gamified one and (Buckley & Clarke, 2018; Sherif et al., 2020) also stated that the gamified group performed better. (Clarke et al., 2017; Costa & Oliveira, 2020; Elgrably & Oliveira, 2018; Fraser et al., 2019; Fu & Clarke, 2016; Sheth et al., 2015, 2013; Valle et al., 2017) did not make a comparison with a control group, but they observed that gamification had a positive impact on the student performance. In addition, the authors pointed out that the students evaluated the gamification experiences as very positive and they were engaged. (Clarke et al., 2019, 2022) evaluated whether increasing the use of a set of learning and engagement strategies that include gamification, referred to as LESs, improved the student performance. The authors compared the performance of the students minimally exposed to LESs with the performance of the students fully exposed to LESs and reported that the latter performed better. Additionally, (Clarke et al., 2020) evaluated the students’ satisfaction when LESs, improved the student performance. The authors compared the performance of the students minimally exposed to LESs with the performance of the students fully exposed to LESs and reported that the latter performed better. Additionally, (Clarke et al., 2020) evaluated the students’ satisfaction when LESs were used, compared to the approach used in a previous course of their degree, and observed that the students were more satisfied when LESs were used. On the contrary, (Khandelwal et al., 2017), after comparing gamified and non-gamified groups, concluded that there was no impact of gamification on the student performance, despite most of the students being in favor of gamification. Similarly, (Jesus et al., 2020) stated that there was no difference in performance between gamified and non-gamified groups, and non-gamified groups were more engaged despite the fact that gamification attracted more students’ attention. Both works reported short time length experiences and emphasized that more investigation is needed. In addition, (Jesus et al., 2020) indicated that the short duration of their gamification experience was a limitation of their research and a more real scenario would be an entire academic semester, leading to improving engagement and performance in the long-term. Those beliefs are consistent with our findings. We have also found that in the first exercises the level of engagement of the students in the experimental group was higher than that of the students in the control group. In other words, their dropout rates were lower and both the interaction with SQLTest and the participation in all activities of the exercises were higher. In addition, we have noted that this engagement follows a downward trend in the last exercise. Regarding the student performance, the experimental group obtained better results in every exercise, and it also follows a downward trend in both experimental and control groups. As a consequence of the downward trend observed, we faced a second challenge: analyzing whether the gamification impact remained constant over the whole experience or it varied in certain moments. To deal with this challenge, we analyzed the results obtained for the experimental group in each individual exercise. The students in the experimental group spent more days working on the first exercise (the active time is twice as many days as in the second exercise), but they executed more test suites and participated more in the second exercise (44% more test suites executions and 5% more participation rate than in the first exercise). In addition, no student dropped out of the seminar in the first two exercises. The active time of the third exercise was similar to the second one, but the number of test suite executions and the participation rate decreased to values slightly lower than those of the first exercise (14% less test suites executions and 1% less participation rate than in the first exercise). Moreover, the dropout rate increased slightly in the third exercise to 1%. In the last exercise, the engagement decreased a bit more. The students worked half a day less in this exercise than in the second and third ones. Furthermore, the number of test suites executions and the participation rate were 18% and 14% lower, respectively, in the last exercise than in the third one. In addition, the dropout rate was slightly higher in the last exercise (7%). Therefore, the impact of gamification on the student engagement did not remain constant over the whole experience, as students seemed to be more engaged in the second exercise and slightly less engaged in the last one. Regarding their performance, the effectiveness and the effectiveness increase were similar in the first two exercises (just slightly higher in the first exercise). In the third exercise, both the effectiveness and the effectiveness increase were 6%, approximately, lower than in the second exercise. Once again, in the last exercise, both metrics were lower than in the previous exercise. The effectiveness decreased another 5%, whereas the effectiveness increase was reduced by 12%. Therefore, the impact of gamification on the student performance varied slightly toward the last stages of the experience. Our findings do not seem to be in line with the results of (Fraser et al., 2019), which indicated that the students’ performance improved throughout the semester. However, int their approach there was not a gamification experience that involved the whole semester, but the laboratory sessions were gamified independently through several games. As a result, the effects of a long gamification experience was not analyzed in that work. In view of the results obtained in the analysis of the second challenge, which revealed that the impact of gamification was slightly lower in the last exercises, we addressed a third challenge: analyzing whether there was a link between gamification and both extrinsic and intrinsic motivation. To respond to this challenge, we analyzed the behavior of the students, mainly in the last exercise. The slight reduction we observed may be caused by the gamification itself and the lack of benefit perceived by the students at the end of the race. In every exercise, some students worked to achieve 100% of test suite effectiveness in order to obtain the maximum 1,000 distance units given by that exercise, as well as the divine points they used to buy distance increments. These distance increments allowed them to advance faster in the race as they could accumulate more than 1,000 distance units per exercise. Other students did not work to achieve 100% of effectiveness, because they were given enough divine points to buy the distance increments for obtaining at least 1,000 distance units in that exercise. This behavior is more noticeable in the last exercise. So, we observed that the students were more extrinsically motivated by the gamification experience to increase their performance in the first exercises because they received the benefits to advance quickly in the race. Consequently, almost 25% of the students had already achieved the finish line by the time the test improvement activity of the last exercise started. When we analyzed in depth the behavior of the students in the last exercise, we observed that 8% of them did not carry out the test improvement activity, that is the last activity, because they had already reached the finish line, whereas 19% of the students did not participate in this activity maybe because their positions in the Olympic race ranking were too far away from rewarded positions. We also observed that 40% of the students reached the finish line by working on the test improvement activity, but 15% of the students did not complete this activity. Maybe they gave up because it was already impossible for them to reach the rewarded positions. So, some students dropped out of the test improvement activity of the last exercise because they had already reached the finish line and no more gamification benefits could be received, while other students gave up maybe because they were disengaged by the lack of expectations of reaching the rewarded positions. Consequently, this study cannot prove that the students were, finally, intrinsically motivated to carry out all academic activities of the seminars. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Authors: Authors: (1) Raquel Blanco, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (rblanco@uniovi.es); (2) Manuel Trinidad, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (manuel.trinidad@uca.es); (3) María José Suárez-Cabal, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (cabal@uniovi.es); (4) Alejandro Calderón, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (alejandro.calderon@uca.es); (5) Mercedes Ruiz, Software Process Improvement and Formal Methods Research Group, University of Cadiz, Department of Computer Science and Engineering, Cádiz, Spain (mercedes.ruiz@uca.es); (6) Javier Tuya, Software Engineering Research Group, University of Oviedo, Department of Computer Science, Gijón, Spain (tuya@uniovi.es). Editor's note: This is part 5 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 5 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 5 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Editor's note: This is part 5 of 7 of a study detailing attempts by researchers to create effective tests using gamification. Read the rest below. Table of Links Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 5 Discussion 6 Threats to validity 7 Conclusions, Acknowledgements and References Abstract and 1 Introduction Abstract and 1 Introduction 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 2 Gamification and software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 2.1 Software testing 2.2 Gamification 2.3 Gamification in software testing education 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 3 Experiment design and 3.1 Course design 3.2 Participants 3.3 Materials 3.4 Procedure 3.5 Metrics 3.2 Participants 3.2 Participants 3.3 Materials 3.3 Materials 3.4 Procedure 3.4 Procedure 3.5 Metrics 3.5 Metrics 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 4 Results and 4.1 RQ1: Student engagement 4.2 RQ2: Student performance 4.2 RQ2: Student performance 5 Discussion 5 Discussion 6 Threats to validity 6 Threats to validity 7 Conclusions, Acknowledgements and References 7 Conclusions, Acknowledgements and References 5 Discussion The main challenge we addressed in the present study was to measure and evaluate the impact of gamification on student engagement and performance. To deal with this challenge, we defined the metrics presented in Section 3.5 and analyzed the results obtained in the controlled experiment that was carried out. The results show that, overall, gamification contributes to the improvement of the student engagement and performance. These findings are in line with those of most works that reported practical gamification experiences in software testing education. (Buckley & Clarke, 2018; Lorincz et al., 2021; Sherif et al., 2020) compared the results achieved by gamified and non-gamified groups; (Lorincz et al., 2021; al., 2021; Sherif et al., 2020) reported that the engagement was higher in the gamified one and (Buckley & Clarke, 2018; Sherif et al., 2020) also stated that the gamified group performed better. (Clarke et al., 2017; Costa & Oliveira, 2020; Elgrably & Oliveira, 2018; Fraser et al., 2019; Fu & Clarke, 2016; Sheth et al., 2015, 2013; Valle et al., 2017) did not make a comparison with a control group, but they observed that gamification had a positive impact on the student performance. In addition, the authors pointed out that the students evaluated the gamification experiences as very positive and they were engaged. (Clarke et al., 2019, 2022) evaluated whether increasing the use of a set of learning and engagement strategies that include gamification, referred to as LESs, improved the student performance. The authors compared the performance of the students minimally exposed to LESs with the performance of the students fully exposed to LESs and reported that the latter performed better. Additionally, (Clarke et al., 2020) evaluated the students’ satisfaction when LESs, improved the student performance. The authors compared the performance of the students minimally exposed to LESs with the performance of the students fully exposed to LESs and reported that the latter performed better. Additionally, (Clarke et al., 2020) evaluated the students’ satisfaction when LESs were used, compared to the approach used in a previous course of their degree, and observed that the students were more satisfied when LESs were used. On the contrary, (Khandelwal et al., 2017), after comparing gamified and non-gamified groups, concluded that there was no impact of gamification on the student performance, despite most of the students being in favor of gamification. Similarly, (Jesus et al., 2020) stated that there was no difference in performance between gamified and non-gamified groups, and non-gamified groups were more engaged despite the fact that gamification attracted more students’ attention. Both works reported short time length experiences and emphasized that more investigation is needed. In addition, (Jesus et al., 2020) indicated that the short duration of their gamification experience was a limitation of their research and a more real scenario would be an entire academic semester, leading to improving engagement and performance in the long-term. Those beliefs are consistent with our findings. We have also found that in the first exercises the level of engagement of the students in the experimental group was higher than that of the students in the control group. In other words, their dropout rates were lower and both the interaction with SQLTest and the participation in all activities of the exercises were higher. In addition, we have noted that this engagement follows a downward trend in the last exercise. Regarding the student performance, the experimental group obtained better results in every exercise, and it also follows a downward trend in both experimental and control groups. As a consequence of the downward trend observed, we faced a second challenge: analyzing whether the gamification impact remained constant over the whole experience or it varied in certain moments. To deal with this challenge, we analyzed the results obtained for the experimental group in each individual exercise. The students in the experimental group spent more days working on the first exercise (the active time is twice as many days as in the second exercise), but they executed more test suites and participated more in the second exercise (44% more test suites executions and 5% more participation rate than in the first exercise). In addition, no student dropped out of the seminar in the first two exercises. The active time of the third exercise was similar to the second one, but the number of test suite executions and the participation rate decreased to values slightly lower than those of the first exercise (14% less test suites executions and 1% less participation rate than in the first exercise). Moreover, the dropout rate increased slightly in the third exercise to 1%. In the last exercise, the engagement decreased a bit more. The students worked half a day less in this exercise than in the second and third ones. Furthermore, the number of test suites executions and the participation rate were 18% and 14% lower, respectively, in the last exercise than in the third one. In addition, the dropout rate was slightly higher in the last exercise (7%). Therefore, the impact of gamification on the student engagement did not remain constant over the whole experience, as students seemed to be more engaged in the second exercise and slightly less engaged in the last one. Regarding their performance, the effectiveness and the effectiveness increase were similar in the first two exercises (just slightly higher in the first exercise). In the third exercise, both the effectiveness and the effectiveness increase were 6%, approximately, lower than in the second exercise. Once again, in the last exercise, both metrics were lower than in the previous exercise. The effectiveness decreased another 5%, whereas the effectiveness increase was reduced by 12%. Therefore, the impact of gamification on the student performance varied slightly toward the last stages of the experience. Our findings do not seem to be in line with the results of (Fraser et al., 2019), which indicated that the students’ performance improved throughout the semester. However, int their approach there was not a gamification experience that involved the whole semester, but the laboratory sessions were gamified independently through several games. As a result, the effects of a long gamification experience was not analyzed in that work. In view of the results obtained in the analysis of the second challenge, which revealed that the impact of gamification was slightly lower in the last exercises, we addressed a third challenge: analyzing whether there was a link between gamification and both extrinsic and intrinsic motivation. To respond to this challenge, we analyzed the behavior of the students, mainly in the last exercise. The slight reduction we observed may be caused by the gamification itself and the lack of benefit perceived by the students at the end of the race. In every exercise, some students worked to achieve 100% of test suite effectiveness in order to obtain the maximum 1,000 distance units given by that exercise, as well as the divine points they used to buy distance increments. These distance increments allowed them to advance faster in the race as they could accumulate more than 1,000 distance units per exercise. Other students did not work to achieve 100% of effectiveness, because they were given enough divine points to buy the distance increments for obtaining at least 1,000 distance units in that exercise. This behavior is more noticeable in the last exercise. So, we observed that the students were more extrinsically motivated by the gamification experience to increase their performance in the first exercises because they received the benefits to advance quickly in the race. Consequently, almost 25% of the students had already achieved the finish line by the time the test improvement activity of the last exercise started. When we analyzed in depth the behavior of the students in the last exercise, we observed that 8% of them did not carry out the test improvement activity, that is the last activity, because they had already reached the finish line, whereas 19% of the students did not participate in this activity maybe because their positions in the Olympic race ranking were too far away from rewarded positions. We also observed that 40% of the students reached the finish line by working on the test improvement activity, but 15% of the students did not complete this activity. Maybe they gave up because it was already impossible for them to reach the rewarded positions. So, some students dropped out of the test improvement activity of the last exercise because they had already reached the finish line and no more gamification benefits could be received, while other students gave up maybe because they were disengaged by the lack of expectations of reaching the rewarded positions. Consequently, this study cannot prove that the students were, finally, intrinsically motivated to carry out all academic activities of the seminars. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv