IntelliGame in Action: Gamifying JavaScript Unit Tests - Experiment

Table of Links

Abstract and Introduction

Background and Related Work

Implementation

Experiment

Results

Conclusions, Acknowledgement, and References

4 EXPERIMENT

The primary aim of the experiment was to replicate the conditions outlined in the original validation paper [8], adhering to the guidelines set by Jedlitschka et al. [4]. However, as the goal was not to assess the effectiveness of the tool but rather to test its applicability with a different programming language and subjects, we introduced some modifications to the experimental approach.

4.1 Participant Selection

We invited graduate students from the Software Engineering course at Politecnico di Torinoto participate voluntarily in the experiment. Successful participation earned students additional points for their Software Engineering exam, with the assurance that their performance would not impact their bonus points to avoid bias. All participating students possessed prior JavaScript testing experience, as the Software Engineering course extensively covered unit testing using the Jest framework.

The sample of participants in this study comprised 152 individuals, with 85% identifying as male and the remaining 15% as female. Over 90% fell within the age range of 22 to 25, being graduate students. The participants were randomly assigned to one of two groups: treatment and control.

The majority of participants had less than three months of experience with JavaScript, and it is worth noting that the treatment group had less experience compared to the control group with 59% and 47% having less than three months of experience, respectively. Additionally, 96% of participants in both groups had less than three months of experience with Jest. In fact, the majority of participants were introduced to JavaScript and Jest during the course’s lectures.

4.2 Project Selection

Initially, with a shift in the reference programming language, the need arose to select an alternative project, departing from the FixedOrderComparator Java class used in the original study [8].

Our criteria for project selection involved finding a subject that presented both challenge and feasibility within a 150-minute timeframe. We sought a real existing project to capture the complexity 3https://github.com/tsantalis/RefactoringMiner of a real-world scenario, emphasizing documentation, a comprehensive test suite, and modularity. Modularity was crucial, allowing participants to choose among modules, and deciding whether to tackle more complex or simpler functions.

The selection process began by exploring JavaScripting[4], a repository of publicly available JavaScript projects. Given the extensive collection, including popular frameworks like React, React Native, Vue, and Angular, we narrowed down our search to importable lightweight libraries with standardized functionalities. We focused on the miscellaneous category for a subject meeting our criteria.

The first project suiting our needs was Date Fns5 , an open-source project featuring 244 functions related to the date data type. Functions ranged from trivial tasks like determining the order between two dates to more complex operations like formatting data with ISO or RFC representation. The project, widely used (over two million times) and well-maintained by 300+ contributors with over a thousand forks, met our requirements.

The project, however, was developed in TypeScript (TS) rather than JS. Notwithstanding the difference between the two programming languages, given the high project usage and reliability, the full interoperability of JS code in TS and the possibility of directly transposing TS code into JS, was deemed that this language difference was not a reason for exclusion. Moreover, we consider it valuable for the reader to discuss in section 5.2 the problems encountered and the solutions put in place to resolve them.

For these reasons, we used the JS version obtained through transpilation [10]. The seamless interoperability between the two languages facilitated the conversion of functions from TS to JS, made possible with the help of the open-source automatic tool transform[6].

4.3 Experiment Task

We carefully chose a set of 23 functions from the original GitHub Date Fns project, prioritizing simplicity in terms of complexity and lines of code. The aim was to ensure that most students could confidently build and test these functions without feeling overwhelmed by their difficulty

Before assigning the experiment task, we conducted a pilot study with three master’s students nearing graduation. This helped validate the task’s scalability. While no data was collected from them, the pilot ensured proper calibration by adjusting the assignment’s complexity, excluding the three most challenging functions.

The functions’ bodies were removed, leaving the requirements, function names, and parameters intact. The project incorporated complete JavaScript documentation for built-in date functions, eliminating the need for online references. Two verification methods were provided: a Jest configuration with empty test files for participants to complete and run, and a main.js file acting like the Main class in Java, mirroring the original experiment’s setup [8].

Two sessions were held: the first with 60 participants and the second with the remaining individuals. This division allowed potential data discarding from the first day if any issues arose. Participants were split into a control group, tackling tasks without achievements, and a treatment group, completing tasks with the plugin enabled.

Participants from the treatment group were instructed on how the gamification plugin works but were not shown in detail the conditions for unlocking each achievement. These were shown in an overlay box displayed on the screen when hovering the mouse over each achievement of the list.

During the experiment, a custom event logger recorded achievement states after each user interaction. An automated script committed and pushed current implementations and log files to a Git repository every minute.

Following the 150-minute implementation phase, an exit questionnaire featured general questions for both groups. The treatment group received an additional page at the end of the survey inquiring about their plugin experiences, with answers on a five-point Likert scale. An optional free-text question allowed participants to share individual comments and feedback.

[4] https://www.javascripting.com/

[5] https://github.com/date-fns/date-fns

[6] https://github.com/ritz078/transform