Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Methodology and 2.1. Research Questions 2. Methodology and 2.1. Research Questions 2.2. Data Collection 2.2. Data Collection 2.3. Data Labelling 2.3. Data Labelling 2.4. Data Extraction 2.4. Data Extraction 2.5. Data Analysis 2.5. Data Analysis 3. Results and Interpretation and 3.1. Type of Problems (RQ1) 3. Results and Interpretation and 3.1. Type of Problems (RQ1) 3.2. Type of Causes (RQ2) 3.2. Type of Causes (RQ2) 3.3. Type of Solutions (RQ3) 3.3. Type of Solutions (RQ3) 4. Implications 4.1. Implications for the Copilot Users 4.1. Implications for the Copilot Users 4.2. Implications for the Copilot Team 4.2. Implications for the Copilot Team 4.3. Implications for Researchers 4.3. Implications for Researchers 5. Threats to Validity 5. Threats to Validity 6. Related Work 6.1. Evaluating the Quality of Code Generated by Copilot 6.1. Evaluating the Quality of Code Generated by Copilot 6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary 6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary 7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References 7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References 4.3. Implications for Researchers The use of Copilot may alter the coding process and increase the time cost of verifying code suggestions, making code explanation highly important. In our research, INCOMPREHENSIBLE SUGGESTION ranks as the fourth most common Suggestion Content Issue, and the code explanation feature is also frequently requested. Some users mentioned that the code suggestions generated by Copilot are excessively long, resulting in reduced readability. This indicates that when Copilot provides relatively complex suggestions, or when users lack coding experience in a particular domain, understanding the code logic and verifying its correctness can be time-consuming. In that case, users may refrain from adopting code suggestions of Copilot as it makes them feel a loss of control over the coding task, according to the research of Vaithilingam et al. (2022). The study by Wang et al. (2023b) also shows that using AI-generated code can lead to substantial review pressure. Therefore, we believe that AI coding tools like Copilot will change the allocation of time spent on various tasks in software development. As expected by some users, the code explanation feature would be helpful. GitHub Copilot Chat now is generally available for organizations and individuals (Zhao, 2023), which is powered by GPT-4. The chat feature enables Copilot to interact more flexibly with users, providing the potential for implementing functions such as code explanation, code refactoring, and vulnerability detection. For future research, we plan to explore how the chat feature can influence the programming process and enhance efficiency of coding tasks. The use of Copilot may alter the coding process and increase the time cost of verifying code suggestions, making code explanation highly important. To accurately gauge user satisfaction with Copilot, it is essential to consider programming tasks across various application domains, as well as the purposes for which users employ Copilot. We identified relatively few Suggestion Content Issues, and one potential reason is that users are less inclined to report specific code-related issues to public Q&A platforms, indicating a certain degree of tolerance for AI-generated erroneous suggestions (Weisz et al., 2021). Therefore, our study results may not fully reflect user satisfaction with the Copilot generated code content. Users often propose new Feature Requests based on their individual needs and coding habits, coupled with their experience of using Copilot. Some of these requests are closely associated with the domains of the applications that Copilot users develop, such as front-end web development and game development. Consequently, user satisfaction levels with Copilot may vary across different application domains. Thus, assessing the capabilities of Copilot through the lens of single programming tasks, such as addressing algorithmic problems or writing SQL statements, may not represent the actual user satisfaction in other application domains. Furthermore, our previous research found that users have diverse objectives when using Copilot (Zhang et al., 2023), directly influencing their expectations. For instance, users looking to quickly grasp new technologies with the assistance of Copilot are more concerned with whether the code suggestions provide helpful programming guidance for subsequent steps; conversely, those who wish to use Copilot for repetitive coding tasks, such as CRUD operations on a database, are more concerned with whether Copilot can generate correct code to reduce the significant effort they spend on fixing similar issues in code suggestions. Therefore, the intended utilization of Copilot also stands as a critical factor influencing the satisfaction of users with Copilot. A focused study on user satisfaction with Copilot based on these two factors will provide an insightful assessment. To accurately gauge user satisfaction with Copilot, it is essential to consider programming tasks across various application domains, as well as the purposes for which users employ Copilot. Authors:
(1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China (xiyuzhou@whu.edu.cn);
(2) Peng Liang (Corresponding Author), School of Computer Science, Wuhan University, Wuhan, China (liangp@whu.edu.cn);
(3) Beiqi Zhang, School of Computer Science, Wuhan University, Wuhan, China (zhangbeiqi@whu.edu.cn);
(4) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China (zengyangli@ccnu.edu.cn);
(5) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany (ahmad.aakash@gmail.com);
(6) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia (mojtaba.shahin@rmit.edu.au);
(7) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland (muhammad.m.waseem@jyu.fi). Authors: Authors: (1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China (xiyuzhou@whu.edu.cn); (2) Peng Liang (Corresponding Author), School of Computer Science, Wuhan University, Wuhan, China (liangp@whu.edu.cn); (3) Beiqi Zhang, School of Computer Science, Wuhan University, Wuhan, China (zhangbeiqi@whu.edu.cn); (4) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China (zengyangli@ccnu.edu.cn); (5) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany (ahmad.aakash@gmail.com); (6) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia (mojtaba.shahin@rmit.edu.au); (7) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland (muhammad.m.waseem@jyu.fi). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

How GitHub and Stack Overflow Data Were Verified for Research Accuracy

How Copilot Can Better Serve Developers

Pair Programming AI Agent 

Too Long; Didn't Read

Make resilience your competitive advantage

Exploring User Needs and Satisfaction with GitHub Copilot

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

11 Proven Solutions to Common GitHub Copilot Problems

Customize ChatGPT for Coding: GPTutor Gives Developers Full Control in VS Code

GPTutor Lets Developers Fine-Tune AI Coding Help Inside VS Code

Developer Productivity with GitHub Copilot & AI Tools by Aditya Mishra

What Developers Really Think About GitHub Copilot

11 Proven Solutions to Common GitHub Copilot Problems

Customize ChatGPT for Coding: GPTutor Gives Developers Full Control in VS Code

GPTutor Lets Developers Fine-Tune AI Coding Help Inside VS Code

Developer Productivity with GitHub Copilot & AI Tools by Aditya Mishra

What Developers Really Think About GitHub Copilot

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps