Authors:
(1) Bo Wang, Beijing Jiaotong University, Beijing, China (wangbo_cs@bjtu.edu.cn);
(2) Mingda Chen, Beijing Jiaotong University, Beijing, China (23120337@bjtu.edu.cn);
(3) Youfang Lin, Beijing Jiaotong University, Beijing, China (yflin@bjtu.edu.cn);
(4) Mike Papadakis, University of Luxembourg, Luxembourg (michail.papadakis@uni.lu);
(5) Jie M. Zhang, King’s College London, London, UK (jie.zhang@kcl.ac.uk). Authors: Authors: (1) Bo Wang, Beijing Jiaotong University, Beijing, China (wangbo_cs@bjtu.edu.cn); (2) Mingda Chen, Beijing Jiaotong University, Beijing, China (23120337@bjtu.edu.cn); (3) Youfang Lin, Beijing Jiaotong University, Beijing, China (yflin@bjtu.edu.cn); (4) Mike Papadakis, University of Luxembourg, Luxembourg (michail.papadakis@uni.lu); (5) Jie M. Zhang, King’s College London, London, UK (jie.zhang@kcl.ac.uk). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 1 Introduction 2 Background and Related Work 2 Background and Related Work 2 Background and Related Work 3 Study Design 3 Study Design 3.1 Overview and Research Questions 3.1 Overview and Research Questions 3.2 Datasets 3.2 Datasets 3.3 Mutation Generation via LLMs 3.3 Mutation Generation via LLMs 3.4 Evaluation Metrics 3.4 Evaluation Metrics 3.5 Experiment Settings 3.5 Experiment Settings 4 Evaluation Results 4 Evaluation Results 4.1 RQ1: Performance on Cost and Usability 4.1 RQ1: Performance on Cost and Usability 4.2 RQ2: Behavior Similarity 4.2 RQ2: Behavior Similarity 4.3 RQ3: Impacts of Different Prompts 4.3 RQ3: Impacts of Different Prompts 4.4 RQ4: Impacts of Different LLMs 4.4 RQ4: Impacts of Different LLMs 4.5 RQ5: Root Causes and Error Types of Non-Compilable Mutations 4.5 RQ5: Root Causes and Error Types of Non-Compilable Mutations 5 Discussion 5 Discussion 5.1 Sensitivity to Chosen Experiment Settings 5.1 Sensitivity to Chosen Experiment Settings 5.2 Implications 5.2 Implications 5.3 Threats to Validity 5.3 Threats to Validity 6 Conclusion and References 6 Conclusion and References 6 Conclusion and References 5 DISCUSSION 5.1 Sensitivity to Chosen Experiment Settings 5.1.1 Experiment Setup of the Context Length. To compare the similarity to real defects, we deliberately select the context around the bug locations for generating mutations for all the approaches. To check the impact of context selection on performance, we conduct experiments on 60 bugs from Defects4J, comparing how different context lengths affect performance. In our previous experiments, the context length is set to three lines around the bug location, so we conduct experiments to compare the effects of two-line and one-line contexts, as shown in Table 9. To measure their similarity in performance, we use the Spearman coefficient and Pearson correlation, as shown in Table 10. We can see that their mutual similarity exceeds 0.95, which is above the typical threshold of 0.85 [40, 51, 69], and the 𝑝-values are all below 0.05. Different context lengths perform similarly, thus validating the setup of our experiment. 5.1.1 Experiment Setup of the Context Length. 5.1.2 Impact of Different Few-Shot Examples. To measure the impact of different few-shot examples on the results, we conduct comparative experiments with various examples. First, we randomly select 3, 6, and 9 additional examples from QuixBugs, and ensure there is no overlap with the default 6 examples. Additionally, despite the risk of data leakage, we select 6 examples from the Defects4J dataset. The results are presented in Table 11. 5.1.2 Impact of Different Few-Shot Examples. We further compute their similarity, shown as Table 12. We can find that their similarity exceeds 0.95, significantly exceeding the typical threshold of 0.85, and the 𝑝-values are all below 0.05. Therefore, we can conclude the performance of different prompts is very similar. 5.1.3 Performance of Using the Same Number of Mutations. In Section 4, we did not restrict the number of mutations generated by each approach. To study the performance of each approach under fixed quantity conditions, we followed the setting of existing study [28, 70] and limited the number of mutations generated by all methods to the minimum produced by 𝜇Bert [15], which is 16,785. For counts exceeding this number, we randomly selected the specified number of mutations for analysis. We conducted 10 rounds of random sampling comparisons, and the results are presented in Table 13. Therefore, under fixed quantity conditions, the mutations generated by GPT-3.5-Turbo are still the closest to real bugs in behavior. 5.1.3 Performance of Using the Same Number of Mutations. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

How Large Language Models Improve Mutation Testing—and What Still Needs Work

Evaluating GPT and Open-Source Models on Code Mutation Tasks

Mutation.TECH

Too Long; Didn't Read

Make resilience your competitive advantage

Sensitivity Analysis of Experiment Parameters in Bug Mutation Studies

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

A Comprehensive Comparison of Tools for Fitting Mutational Signatures

Study Finds AI Code Mutations Help Developers Catch Bugs Faster

Mutation Testing with GPT and CodeLlama

We Designed a Study to See If AI Can Imitate Real Software Bugs

Using LLMs to Mutate Java Code

A Comprehensive Comparison of Tools for Fitting Mutational Signatures

Study Finds AI Code Mutations Help Developers Catch Bugs Faster

Mutation Testing with GPT and CodeLlama

We Designed a Study to See If AI Can Imitate Real Software Bugs

Using LLMs to Mutate Java Code

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps