Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Prior conceptualisations of intelligent assistance for programmers 2. Prior conceptualisations of intelligent assistance for programmers 3. A brief overview of large language models for code generation 3. A brief overview of large language models for code generation 4. Commercial programming tools that use large language models 4. Commercial programming tools that use large language models 5. Reliability, safety, and security implications of code-generating AI models 5. Reliability, safety, and security implications of code-generating AI models 6. Usability and design studies of AI-assisted programming 6. Usability and design studies of AI-assisted programming 7. Experience reports and 7.1. Writing effective prompts is hard 7. Experience reports and 7.1. Writing effective prompts is hard 7.2. The activity of programming shifts towards checking and unfamiliar debugging 7.2. The activity of programming shifts towards checking and unfamiliar debugging 7.3. These tools are useful for boilerplate and code reuse 7.3. These tools are useful for boilerplate and code reuse 8. The inadequacy of existing metaphors for AI-assisted programming 8.1. AI assistance as search 8.1. AI assistance as search 8.2. AI assistance as compilation 8.2. AI assistance as compilation 8.3. AI assistance as pair programming 8.3. AI assistance as pair programming 8.4. A distinct way of programming 8.4. A distinct way of programming 9. Issues with application to end-user programming 9. Issues with application to end-user programming 9.1. Issue 1: Intent specification, problem decomposition and computational thinking 9.1. Issue 1: Intent specification, problem decomposition and computational thinking 9.2. Issue 2: Code correctness, quality and (over)confidence 9.2. Issue 2: Code correctness, quality and (over)confidence 9.3. Issue 3: Code comprehension and maintenance 9.3. Issue 3: Code comprehension and maintenance 9.4. Issue 4: Consequences of automation in end-user programming 9.4. Issue 4: Consequences of automation in end-user programming 9.5. Issue 5: No code, and the dilemma of the direct answer 9.5. Issue 5: No code, and the dilemma of the direct answer 10. Conclusion 10. Conclusion A. Experience report sources A. Experience report sources References References 4. Commercial programming tools that use large language models OpenAI Codex is a version of GPT that is fine-tuned on publicly available source code (Chen, Tworek, Jun, Yuan, de Oliveira Pinto, et al., 2021). While Codex itself is not a programmer-facing tool, OpenAI has commercialised it in the form of an API that can be built upon. The principal commercial implementation of Codex thus far has been in Github Copilot.[5] Copilot is an extension that can be installed into code editors such as Neovim, Jetbrains, and Visual Studio Code. Copilot uses Codex, drawing upon the contents of the file being edited, related files in the project, and file paths or URLs of repositories. When triggered, it generates code at the cursor location, in much the same way as autocomplete. To help expand developer expectations for the capabilities of Copilot beyond the previous standard uses of autocomplete, suggested usage idioms for Copilot include: writing a comment explaining what a function does, and the function signature, and allowing Copilot to complete the function body; completing boilerplate code; and defining test cases (Figures 1 and 5). Programmers can cycle between different generations from the model, and once a particular completion has been accepted it can be edited like any other code. As of 23 June 2022, Amazon has announced a Copilot-like feature called CodeWhisperer,[6] which also applies a large language model trained on a corpus of source code to generate autocompletions based on comments and code. The marketing material describes a set of safety features, such as: detecting when generated code is similar to code in the training set, detecting known security vulnerabilities in the generated code, and “removing code that may be considered biased and unfair” (although this latter claim induces skepticism). At present CodeWhisperer is not widely available and thus little is known of its use in practice. Other commercial implementations of AI-assisted autocompletion features include Visual Studio Intellicode (Silver, 2018) (Figure 4) and Tabnine (Figure 3) [7]. These are more limited in scope than Copilot and their user experience is commensurable to that of using ‘traditional’ autocomplete, i.e., autocomplete that is driven by static analysis, syntax, and heuristics.[8].The structure of the machine learning model used by these implementations is not publicly disclosed; however, both rely on models that have been trained on large corpora of publicly available source code. It is interesting to note, that despite the wide variety of types of intelligent programmer assistance we have discussed in Section 2 for several aspects of programming (authoring, transcription, modification, debugging, and learning), commercial implementations of assistance based on large language models thus far are aimed primarily at authoring. Authoring can be viewed as the first natural application of a generative language model, but the programming knowledge in these models can of course be used for assisting programmers in other activities, too. Authors:
(1) Advait Sarkar, Microsoft Research, University of Cambridge (advait@microsoft.com);
(2) Andrew D. Gordon, Microsoft Research, University of Edinburgh (adg@microsoft.com);
(3) Carina Negreanu, Microsoft Research (cnegreanu@microsoft.com);
(4) Christian Poelitz, Microsoft Research (cpoelitz@microsoft.com);
(5) Sruti Srinivasa Ragavan, Microsoft Research (a-srutis@microsoft.com);
(6) Ben Zorn, Microsoft Research (ben.zorn@microsoft.com). Authors: Authors: (1) Advait Sarkar, Microsoft Research, University of Cambridge (advait@microsoft.com); (2) Andrew D. Gordon, Microsoft Research, University of Edinburgh (adg@microsoft.com); (3) Carina Negreanu, Microsoft Research (cnegreanu@microsoft.com); (4) Christian Poelitz, Microsoft Research (cpoelitz@microsoft.com); (5) Sruti Srinivasa Ragavan, Microsoft Research (a-srutis@microsoft.com); (6) Ben Zorn, Microsoft Research (ben.zorn@microsoft.com). This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv available on arxiv [5] https://copilot.github.com/ [6] https://aws.amazon.com/codewhisperer/features/ [7] https://www.tabnine.com/ [8] As of 15 June 2022, Tabnine has announced a shift to language model-driven autocompletion that more closely resembles the abilities of Copilot (Weiss, 2022).

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Amazon

OpenAI

The Dark Side Of AI: Reliability, Safety, And Security In Code Generation

A Quick Guide To LLM Code Generation Technology And Its Limits

Pair Programming AI Agent 

Too Long; Didn't Read

Make resilience your competitive advantage

GitHub Copilot Leads The Charge In Commercial LLM-Assisted Programming

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

11 Proven Solutions to Common GitHub Copilot Problems

Human-Human vs Human-AI Pair Programming

The Pros and Cons of Pair Programming

What Makes Pair Programming a Success (or a Failure)?

Breaking Down the Human-AI Pair Programming Experience

11 Proven Solutions to Common GitHub Copilot Problems

Human-Human vs Human-AI Pair Programming

The Pros and Cons of Pair Programming

What Makes Pair Programming a Success (or a Failure)?

Breaking Down the Human-AI Pair Programming Experience

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps