Insights into Copilot Usage: A Comprehensive Analysis from Real User Issues

VI. RELATED WORK

Several studies focused on investigating the performance of GitHub Copilot in actual software development, as well as the opinions of software practitioners on it. Wang et al. [22] conducted an interview with 15 practitioners and surveyed 599 practitioners from 18 IT companies regarding their expectations of code auto-completion. They found that 13% of the participants had used Copilot. Jaworski et al. [23] prepared a survey questionnaire consisting of 18 questions to investigate developers’ attitudes towards Copilot. The research findings indicate that most people have a positive attitude towards the tool, but few participants are willing to use it, and there are concerns about security issues associated with using Copilot. Imai et al. [9] conducted experiments with 21 participants to compare the effectiveness of GitHub Copilot paired with human programmers in terms of productivity and code quality. The results indicate that while Copilot can increase productivity by adding more lines of code, the generated code quality is lower due to the need to subsequently remove more lines of code during testing. Barke et al. [10] observed 20 participants collaborating with Copilot to complete different programming tasks in four languages, and found that the interaction with the programming assistant was bimodal. Bird et al. [20] conducted three studies aimed at understanding how developers utilize Copilot. Their findings suggest that developers spent a lot of time assessing the suggestions generated by Copilot instead of completing the task without the tool. Peng et al. [11] presented the results of a controlled experiment using GitHub Copilot as an AI collaborative programmer. They found that the experimental group who had access to the AI collaborative programmer completed tasks 55.8% faster than the control group. Zhang et al. [19] investigated the programming languages, IDEs, associated technologies, implemented functionalities, advantages, limitations, and challenges when utilizing Copilot. Liang et al. [24] conducted a survey among a large number of developers and found that the primary motivation for developers to use AI programming assistants is to reduce keystrokes, complete programming tasks quickly, and recall syntax. However, the impact of using these tools to help generate potential solutions is not significant. Sandoval et al. [25] analyzed the use of Copilot for programming and compared it to earlier forms of programmer assistance. They also explored potential challenges that could arise when applying LLMs to programming.

In contrast to the prior studies, our research distinguishes itself by gathering comprehensive data from GitHub Issues, Discussions, and SO posts. Specifically, our study delves into the issues confronted by users during the practical use of Copilot, exploring both the underlying causes and potential solutions associated with these issues.