IntelliGame in Action: Gamifying JavaScript Unit Tests - Abstract and Intro

Table of Links

Abstract and Introduction

Background and Related Work

Implementation

Experiment

Results

Conclusions, Acknowledgement, and References

ABSTRACT

This paper investigates the integration and assessment of IntelliGame, a gamification plugin initially designed for Java development, within the realm of JavaScript unit testing. We aim to verify the generalizability of IntelliGame to JavaScript development and to provide valuable insights into the experiment’s design. For this, we first customize IntelliGame for JavaScript, and then conduct a controlled experiment involving 152 participants utilizing the Jest testing framework, and finally examine its influence on testing behavior and the overall developer experience. The findings from this study provide valuable insights for improving JavaScript testing methodologies through the incorporation of gamification.

CCS CONCEPTS

• Software and its engineering → Software testing and debugging; Integrated and visual development environments.

KEYWORDS

Gamification, IDE, IntelliJ, Software Testing

ACM Reference Format:

Philipp Straubinger, Tommaso Fulcini, Gordon Fraser, and Marco Torchiano. 2024. IntelliGame in Action: An Experience Report on Gamifying JavaScript Unit Tests. In 2024 First IDE Workshop (IDE ’24), April 20, 2024, Lisbon, Portugal. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3643796. 3648466

1 INTRODUCTION

IntelliGame [8] is a plugin for the popular IntelliJ IDEA, which enables gamification covering various aspects of software testing, such as test execution, coverage evaluation, debugging, and test refactoring. It introduces 27 different achievements, each with incremental levels, providing positive feedback when developers exhibit commendable testing behavior. IntelliGame provides real-time analysis of developer interactions, notifications, and a user interface within IntelliJ to display achievements and progress.

In its original evaluation [8], we conducted a controlled experiment with 49 participants to study the impact of IntelliGame on testing behavior. The experiment involved a Java programming task with participants divided into different groups, aiming to assess the impact of the gamified environment on testing behavior, test suites, achievement levels, code functionality, and the overall developer experience. The results showed clearly that IntelliGame had a positive effect on testing behavior. Participants using IntelliGame wrote more tests, achieved higher code coverage and mutation scores, ran tests more frequently, and implemented functionality earlier compared to the control group. The impact on resulting test suites was substantial, and achievements correlated positively with various testing metrics.

The goal of this paper is to build upon these promising results achieved with IntelliGame for Java environments, and to assess whether these outcomes can be replicated on a larger scale and in a different context. In particular, the extension of the positive impact of achievements in the IDE to various programming languages remains unexplored. Thus we shifted the focus of this study to JavaScript (JS), recognized as the most used language as of 2023 [6]. With the growing importance of JS, there is an increasing demand to instruct and practice language-specific testing activities. Expanding beyond the previous sample, this study engages a broader audience with multiple tasks, allowing for more extensive code development and testing. This expansion, coupled with the shift to a different programming language, considers both generalizability and effectiveness beyond a short-term context.

The objective of this paper is to provide an in-depth description of the experiment’s design, including challenges encountered, lessons learned, and recommendations for future iterations of JSbased experiments. Notably, there is a gap in the existing literature regarding programming experiments centered on an open-source JavaScript target [9], emphasizing a fully replicable and systematic approach. In Section 2 we provide some background about the topic of the paper with also some examples of related works, inSection 3 the implementation approach for the JavaScript version of IntelliGame is described, while Section 4 shows the selection process for the project as a subject is meticulously detailed, shedding light on the rationale behind the choice.