Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Methodology and 2.1. Research Questions 2. Methodology and 2.1. Research Questions 2.2. Data Collection 2.2. Data Collection 2.3. Data Labelling 2.3. Data Labelling 2.4. Data Extraction 2.4. Data Extraction 2.5. Data Analysis 2.5. Data Analysis 3. Results and Interpretation and 3.1. Type of Problems (RQ1) 3. Results and Interpretation and 3.1. Type of Problems (RQ1) 3.2. Type of Causes (RQ2) 3.2. Type of Causes (RQ2) 3.3. Type of Solutions (RQ3) 3.3. Type of Solutions (RQ3) 4. Implications 4.1. Implications for the Copilot Users 4.1. Implications for the Copilot Users 4.2. Implications for the Copilot Team 4.2. Implications for the Copilot Team 4.3. Implications for Researchers 4.3. Implications for Researchers 5. Threats to Validity 5. Threats to Validity 6. Related Work 6.1. Evaluating the Quality of Code Generated by Copilot 6.1. Evaluating the Quality of Code Generated by Copilot 6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary 6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary 7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References 7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References 4. Implications 4.1. Implications for the Copilot Users Seeking inspiration from Copilot’s code suggestions rather than relying on them can lead to better control over your coding tasks. In Feature Request, we have observed that users frequently present their wish to accept code suggestions word-by-word or line-by-line when reviewing them. As a user complained in an SO post, “Copilot inputs tons of code when people just want a tiny part” (SO #74091857). Users’ desire for such requests may reflect their concerns about losing control over their coding tasks because Copilot sometimes is too “productive” for them. In the experiment conducted by Vaithilingam et al. (2022), five participants perceived the same worry when they failed to understand the code suggested by Copilot. Thus, finding a right balance between leveraging Copilot’s productivity and maintaining developers’ command over their code is important for Copilot users. As pointed out by Ziegler et al. (2024), the value of Copilot lies in whether the suggestions serve as a useful starting point for further development. Therefore, a useful position of Copilot would be a tool that provides coding insights, rather than a complete replacement for developers in coding tasks. Before using Copilot for coding, users should ideally have a clear understanding of their code structure. Then, by guiding Copilot to provide code suggestions, they can further optimize the code implementation. This way ensures that users maintain control over their code while incorporating the inspiring suggestions provided by Copilot, keeping the role of users as a real “pilot” of the coding tasks. Seeking inspiration from Copilot’s code suggestions rather than relying on them can lead to better control over your coding tasks. Review the code suggestions provided by Copilot before accepting them to prevent introducing quality issues into your project. According to the results of RQ1, the emergence of INSECURE SUGGESTION, LESS EFFICIENT SUGGESTION, and SUGGESTION WITH INVALID SYNTAX indicates potential quality issues in the code suggested by Copilot. The code quality of Copilot generated code has been a critical topic for the Copilot team and many researchers (e.g., Yetistiren et al. (2022), Siddiq et al. (2022)). Since the outputs of LLMs are unpredictable, it is unrealistic to expect Copilot to consistently produce quality-assured code under all circumstances. Additionally, we identified only a few issues related to the quality of Copilot generated code in our study. One possible reason is that developers may lack experience or may not be attentive enough to identifying quality issues such as code vulnerabilities (Fu et al., 2024) and code smells (Zhang et al., 2024) in Copilot generated code. We also noticed that users may find it challenging to fully understand the code suggestions provided by Copilot in INCOMPREHENSIBLE SUGGESTION, leading to difficulties in recognizing the quality issues introduced by Copilot. Therefore, it is essential to carefully examine whether there are any quality issues within the code suggestions from Copilot before accepting them. Review the code suggestions provided by Copilot before accepting them to prevent introducing quality issues into your project. It is advisable to utilize IDEs or code editors that are officially supported by Copilot. According to our study, discussions regarding Copilot usage problems are predominantly reported on mainstream coding development platforms such as VSCode, Visual Studio, and PyCharm. The Copilot team also actively reacts to the queries arising on the platforms officially supported by Copilot. Consequently, users utilizing Copilot on these IDE or code editors are more likely to find solutions for problems like Operation Issues and Compatibility Issuesthrough public Q&A platforms. We found that Unsupported Platform ranks as the fourth most frequent causes leading to Copilot usage problems. Based on the results of RQ2 (see Table 3), it is also one of the primary reasons causing various types of Operation Issue, which suggests that users may encounter unforeseen problems when attempting to use Copilot on the platforms that are not supported yet. These problems are often challenging to resolve. In our study, none of the 31 issues associated with Unsupported Platform has been resolved effectively. Therefore, users should prioritize the use of Copilot on officially supported platforms to ensure an optimal experience. It is advisable to utilize IDEs or code editors that are officially supported by Copilot. Authors:
(1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China (xiyuzhou@whu.edu.cn);
(2) Peng Liang (Corresponding Author), School of Computer Science, Wuhan University, Wuhan, China (liangp@whu.edu.cn);
(3) Beiqi Zhang, School of Computer Science, Wuhan University, Wuhan, China (zhangbeiqi@whu.edu.cn);
(4) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China (zengyangli@ccnu.edu.cn);
(5) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany (ahmad.aakash@gmail.com);
(6) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia (mojtaba.shahin@rmit.edu.au);
(7) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland (muhammad.m.waseem@jyu.fi). Authors: Authors: (1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China (xiyuzhou@whu.edu.cn); (2) Peng Liang (Corresponding Author), School of Computer Science, Wuhan University, Wuhan, China (liangp@whu.edu.cn); (3) Beiqi Zhang, School of Computer Science, Wuhan University, Wuhan, China (zhangbeiqi@whu.edu.cn); (4) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China (zengyangli@ccnu.edu.cn); (5) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany (ahmad.aakash@gmail.com); (6) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia (mojtaba.shahin@rmit.edu.au); (7) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland (muhammad.m.waseem@jyu.fi). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

How Developers Stay in Control While Using Copilot

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

11 Proven Solutions to Common GitHub Copilot Problems

Customize ChatGPT for Coding: GPTutor Gives Developers Full Control in VS Code

GPTutor Lets Developers Fine-Tune AI Coding Help Inside VS Code

Developer Productivity with GitHub Copilot & AI Tools by Aditya Mishra

What Developers Really Think About GitHub Copilot

11 Proven Solutions to Common GitHub Copilot Problems

Customize ChatGPT for Coding: GPTutor Gives Developers Full Control in VS Code

GPTutor Lets Developers Fine-Tune AI Coding Help Inside VS Code

Developer Productivity with GitHub Copilot & AI Tools by Aditya Mishra

What Developers Really Think About GitHub Copilot

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps