Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Methodology and 2.1. Research Questions 2. Methodology and 2.1. Research Questions 2.2. Data Collection 2.2. Data Collection 2.3. Data Labelling 2.3. Data Labelling 2.4. Data Extraction 2.4. Data Extraction 2.5. Data Analysis 2.5. Data Analysis 3. Results and Interpretation and 3.1. Type of Problems (RQ1) 3. Results and Interpretation and 3.1. Type of Problems (RQ1) 3.2. Type of Causes (RQ2) 3.2. Type of Causes (RQ2) 3.3. Type of Solutions (RQ3) 3.3. Type of Solutions (RQ3) 4. Implications 4.1. Implications for the Copilot Users 4.1. Implications for the Copilot Users 4.2. Implications for the Copilot Team 4.2. Implications for the Copilot Team 4.3. Implications for Researchers 4.3. Implications for Researchers 5. Threats to Validity 5. Threats to Validity 6. Related Work 6.1. Evaluating the Quality of Code Generated by Copilot 6.1. Evaluating the Quality of Code Generated by Copilot 6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary 6.2. Copilot’s Impact on Practical Development and 6.3. Conclusive Summary 7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References 7. Conclusions, Data availability, Acknowledgments, CRediT authorship contribution statement and References 3.3. Type of Solutions (RQ3) 3.3.1. Results 3.3.1. Results 3.3.1. Results As mentioned in Section 2.4.2, not all problems have corresponding solutions that can be extracted. As a result, we identified a total of 497 solutions, which were used to address 36.7% of all problems, and categorized into 11 types as shown in Table 4. The result reveals that most of the usage bugs were addressed by Bug fixed by Copilot (27.2%). When users tried to solve problems themselves, Modify Configuration/Setting (22.1%), Use Suitable Version (17.1%), and Reinstall/Restart/Reauthorize Copilot (12.1%) were commonly used as effective solutions. The example, count, and proportion of each type of solution are presented in Table 4. Modify Configuration/Setting Use Suitable Version Reinstall/Restart/Reauthorize Copilot • Bug Fixed by Copilot (BFC) refers to the process where the Copilot team collects problems reported by users or conducts functionality test on Copilot, and then addresses identified bugs. For example, a user failed to use Copilot because of the “problem from Copilot’s server-side”, and this problem was fixed by the Copilot team as soon as they noticed it (Issue #86). Typically, the resolution process to the problems by the Copilot team is unclear for users. • Bug Fixed by Copilot • Modify Configuration/Setting (MCS) refers to the solutions that users adjust settings or configuration files of Copilot to solve specific usage problems. For example, by “updating the file setting.json”, a user resolved the problem of Copilot “not adding closing brackets” (SO #68347605). • Modify Configuration/Setting • Use Suitable Version (USV) refers to users choosing a suitable version of Copilot that resolves the problems faced in the Copilot version currently used. This type of solution also includes adjusting the version of IDEs or code editors to be compatible with Copilot. For example, a user who wanted to use Copilot in Visual Studio solved the problem of “not being able to find the GitHub copilot extension”, by “upgrading Visual Studio to 17.2” (Discussion #18566). • Use Suitable Version • Reinstall/Restart/Reauthorize Copilot (RC) is a type of solution allowing users to reset Copilot to its original state, thereby resolving any previous errors or restoring settings to their default conditions. For example, a user found that “GitHub Copilot could not connect to server” in VSCode, and solved it by simply “restarting extension” (Discussion #27378). • Reinstall/Restart/Reauthorize Copilot • Feature Implemented by Copilot (FIC) refers to the introduction or enhancement of a feature by the Copilot team. For example, the latest Copilot “contains preliminary support for connecting through HTTP proxy servers”, addressing the AUTHENTICATION FAILURE previously experienced by some users (SO #73242748). • Feature Implemented by Copilot • Follow Official Instruction (FOI) refers to the process where users follow the steps provided by the Copilot UI or described in the user manuals during registration, login, subscription, and configuration. This type of solution usually aimed at assisting users who are not familiar with Copilot. For example, a user was advised to consult the “Copilot doc” for guidance on configuring keyboard shortcuts on Mac (SO #75276040). • Follow Official Instruction • Disable Interfering Factors (DIF) refers to the process where users identify and remove the factors disrupting the normal operation of Copilot, such as other activated plug-ins and HTTP proxies that could interfere with network connection. For example, a user reported that “GitHub Copilot autosuggesions are not autofilled in .md files”, and solved this problem by disabling the “Markdown all in one” plug-in (Discussion #10203). • Disable Interfering Factors • Restart Runtime Environment (RRE) refers to the process of restarting the environment Copilot operates on to resolve any errors affecting the current runtime environment. For example, restarting the computer and IDE on which Copilot is running helped a user solve an INSTALLATION FAILURE on IntelliJ IDEA 2022 (Discussion #17638). • Restart Runtime Environment • Modify the Input Way (MIW) refers to users changing the way they prompt Copilot to generate code, to address problems that Copilot does not automatically provide code suggestions, or to make Copilot provide codes that better meet their expectations. For example, “going to a new line after a code comment, and starting typing what you want to create” resolved the problem that Copilot created “the same comment over and over again” (SO #70995718). • Modify the Input Way • Install/Update Framework (IUF) refers to installing or updating the frameworks in IDEs and code editors for Copilot to operate. This solution is primarily employed to resolve the incompatibility between Copilot.vim (Pope, 2024) and Node.js. For example, a user found that he had to “install node.js” for Copilot to work in Neovim (Discussion #40300). • Install/Update Framework • Others: Different from Copilot usage problems and their causes, which can be categorized into specific categories and types, several dedicated solutions aimed at addressing certain Copilot usage problems are classified under the category Others. For example, using the VSCode extension “Win-CA” in append mode to resolve ACCESSING FAILURE of Copilot was identified only once, making it difficult to form a new type of solution (SO #71367058). Others 3.3.2. Solutions to Problems Mapping 3.3.2. Solutions to Problems Mapping 3.3.2. Solutions to Problems Mapping Table 5 illustrates the mapping relationship of Copilot related problems to their solution types, using abbreviations to represent each type of solution. For example, “BFC” represents Bug Fixed by Copilot. The full names for all types of solutions are provided in the note of Table 5. For Operation Issue, 44.5% of the cases are effectively addressed. Specifically, ACCESSING FAILURE is mainly addressed by BFC; AUTHENTICATION FAILURE is primarily resolved by BFC, MCS, and RC; FUNCTIONALITY FAILURE is commonly fixed by BFC, MCS, and USV; and STARTUP ISSUE is mostly solved by BFC, MCS, USV, and RC. To address INSTALLATION ISSUE and VERSION CONTROL ISSUE, USV is the primarily employed solution. Operation Issue For Compatibility Issue, effective solutions were identified in 39.3% of the cases. BFC, MCS, and USV are the commonly used solutions for resolving EDITOR/IDE COMPATIBILITY ISSUE, while BFC, MCS, and USV are mostly employed for addressing PLUG-IN COMPATIBILITY ISSUE. Moreover, resolving KEYBOARD COMPATIBILITY ISSUE mainly relies on MCS. Compatibility Issue For Feature Request, 21.7% of the cases have effective solutions. Specifically, FUNCTION REQUEST is primarily addressed by MCS and FIC. Two cases of UI REQUEST were addressed by MCS. In addition, although the solutions for PROFESSIONAL COPILOT VERSION have not been identified, GitHub announced the availability of Copilot Enterprise on February 27, 2024 (GitHub, 2024a), potentially addressing the need for a professional version of Copilot. Feature Request We identified effective solutions for 24.1% of User Experience Issues. POOR AUTHENTICATION EXPERIENCE is addressed by BFC, USV, and RC, while POOR FUNCTIONALITY EXPERIENCE is mainly resolved by MCS. Moreover, BFC is the only solution identified for addressing POOR PERFORMANCE. User Experience Issues We only identified five solutions for Suggestion Content Issue, accounting for 10.2% of the total cases. BFC and MIW were employed to address NONSENSICAL SUGGESTION and LOW QUALITY SUGGESTION ISSUES, and no solutions have been identified for resolving INSECURE SUGGESTION, LESS EFFICIENT SUGGESTION, SUGGESTION WITH BUGS, INCOMPREHENSIBLE SUGGESTION, and SUGGESTION WITH INVALID SYNTAX. Suggestion Content Issue Only 6.7% of Copyright and Policy Issues have corresponding solutions. For CODE COPYRIGHT ISSUE and CODE TELEMETRY ISSUE, MCS is the identified solution. Copyright and Policy Issues 3.3.3. Interpretation 3.3.3. Interpretation 3.3.3. Interpretation Frequency of Solutions: BFC is the most commonly employed solution for addressing problems related to Copilot usage, which is reasonable since Copilot Internal Error is identified as the most common cause of Copilot usage problems, indicating that many problems of Copilot cannot be resolved by users. MCS, USV, and RC are frequently employed solutions when users attempt to resolve Copilot related problems by themselves. Typically, users can obtain relevant experience and knowledge in resolving Copilot Frequency of Solutions: usage problems by these three methods on public Q&A platforms (e.g., GitHub Issues, GitHub Discussions, SO). FIC ranks as the fifth most frequently used solution, reflecting the expansion and improvement of Copilot features to match the requirements of a large user community. The remaining six types of solution account for only 11.8% of the total number of solutions, but can still provide valuable experiences for users who face specific Copilot related problems. For instance, DIF can effectively resolve the conflicts between the plug-in “Markdown All in One” and “Copilot”, saving lots of time for the users who experience the same problem. Mapping of Solutions to Problems: For Operation Issue and Compatibility Issue, a great number of effective solutions have been identified. This is partially due to the high number of Operation Issue and Compatibility Issue, and their direct impact on the proper functioning of Copilot. As a result, both the Copilot team and users are inclined to promptly address these two categories of Copilot related problems. Besides the errors in Copilot server that can be addressed by BFC, some Operation Issues and Compatibility Issues are attributed to the Copilot running environments of users. Therefore, users can address the problems in these two categories through MCS, USV, and RC. The relatively limited number of solutions for Suggestion Content Issue reflects the lack of effective methods for users to adjust the code suggested by Copilot. Feature Request and User Experience Issue usually require a new release of Copilot by the Copilot team to expand and optimize the features of Copilot, making it difficult to meet the expectations of some users in a short term. Additionally, we found that MCS is frequently utilized by users when trying to get the functionality they want without waiting the new release of Copilot. Fewer solutions were identified for Copyright and Policy Issue, indicating that it is important to provide satisfactory solutions to address the concerns of code leakage. Mapping of Solutions to Problems: Authors:
(1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China (xiyuzhou@whu.edu.cn);
(2) Peng Liang (Corresponding Author), School of Computer Science, Wuhan University, Wuhan, China (liangp@whu.edu.cn);
(3) Beiqi Zhang, School of Computer Science, Wuhan University, Wuhan, China (zhangbeiqi@whu.edu.cn);
(4) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China (zengyangli@ccnu.edu.cn);
(5) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany (ahmad.aakash@gmail.com);
(6) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia (mojtaba.shahin@rmit.edu.au);
(7) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland (muhammad.m.waseem@jyu.fi). Authors: Authors: (1) Xiyu Zhou, School of Computer Science, Wuhan University, Wuhan, China (xiyuzhou@whu.edu.cn); (2) Peng Liang (Corresponding Author), School of Computer Science, Wuhan University, Wuhan, China (liangp@whu.edu.cn); (3) Beiqi Zhang, School of Computer Science, Wuhan University, Wuhan, China (zhangbeiqi@whu.edu.cn); (4) Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China (zengyangli@ccnu.edu.cn); (5) Aakash Ahmad, School of Computing and Communications, Lancaster University Leipzig, Leipzig, Germany (ahmad.aakash@gmail.com); (6) Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia (mojtaba.shahin@rmit.edu.au); (7) Muhammad Waseem, Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland (muhammad.m.waseem@jyu.fi). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Huobi Token

How Developers Stay in Control While Using Copilot

The 16 Main Reasons Why GitHub Copilot Breaks

Pair Programming AI Agent 

Too Long; Didn't Read

Make resilience your competitive advantage

11 Proven Solutions to Common GitHub Copilot Problems

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

The Final Verdict on Pair vs. Solo Programming: A Summary

Customize ChatGPT for Coding: GPTutor Gives Developers Full Control in VS Code

GPTutor Lets Developers Fine-Tune AI Coding Help Inside VS Code

Developer Productivity with GitHub Copilot & AI Tools by Aditya Mishra

What Developers Really Think About GitHub Copilot

The Final Verdict on Pair vs. Solo Programming: A Summary

Customize ChatGPT for Coding: GPTutor Gives Developers Full Control in VS Code

GPTutor Lets Developers Fine-Tune AI Coding Help Inside VS Code

Developer Productivity with GitHub Copilot & AI Tools by Aditya Mishra

What Developers Really Think About GitHub Copilot

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps