Microsoft’s AutoDev: The AI That Builds, Tests, and Fixes Code on Its Own

Authors: Michele Tufano (Microsoft, Redmond, USA) Anisha Agarwal (Microsoft, Redmond, USA) Jinu Jang (Microsoft, Redmond, USA) Roshanak Zilouchian (Microsoft, Redmond, USA) Neel Sundaresan (Microsoft, Redmond, USA) Authors: Michele Tufano (Microsoft, Redmond, USA) Anisha Agarwal (Microsoft, Redmond, USA) Jinu Jang (Microsoft, Redmond, USA) Roshanak Zilouchian (Microsoft, Redmond, USA) Neel Sundaresan (Microsoft, Redmond, USA) Michele Tufano (Microsoft, Redmond, USA) Anisha Agarwal (Microsoft, Redmond, USA) Jinu Jang (Microsoft, Redmond, USA) Roshanak Zilouchian (Microsoft, Redmond, USA) Neel Sundaresan (Microsoft, Redmond, USA) Abstract The landscape of software development has witnessed a paradigm shift with the advent of AI-powered assistants, exemplified by GitHub Copilot. However, existing solutions are not leveraging all the potential capabilities available in an IDE such as building, testing, executing code, git operations, etc. Therefore, they are constrained by their limited capabilities, primarily focusing on suggesting code snippets and file manipulation within a chat-based interface. To fill this gap, we present AutoDev, a fully automated AI-driven software development framework, designed for autonomous planning and execution of intricate software engineering tasks. AutoDev enables users to define complex software engineering objectives, which are assigned to AutoDev’s autonomous AI Agents to achieve. These AI agents can perform diverse operations on a codebase, including file editing, retrieval, build processes, execution, testing, and git operations. They also have access to files, compiler output, build and testing logs, static analysis tools, and more. This enables the AI Agents to execute tasks in a fully automated manner with a comprehensive understanding of the contextual information required. Furthermore, AutoDev establishes a secure development environment by confining all operations within Docker containers. This framework incorporates guardrails to ensure user privacy and file security, allowing users to define specific permitted or restricted commands and operations within AutoDev. autonomous In our evaluation, we tested AutoDev on the HumanEval dataset, obtaining promising results with 91.5% and 87.8% of Pass@1 for code generation and test generation respectively, demonstrating its effectiveness in automating software engineering tasks while maintaining a secure and user-controlled development environment. 1 Introduction As developers increasingly adopt AI assistants such as ChatGPT for their development tasks, productivity gains become evident. AI coding assistants have further advanced into integrated development environments (IDEs) like GitHub Copilot [2], where they offer code suggestions both within chat interfaces and directly within files. 2 However, these AI coding assistants, despite their integration into IDEs, exhibit limited functionalities and lack contextual awareness[3, 6]. They often do not leverage all IDE capabilities such as invoking linters, compilers, or executing command-line operations, and consequently, developers still need to manually validate syntax and ensure the correctness of AI-generated code, execute the codebase, and inspect error logs. 3 6 AutoDev bridges this gap by offering autonomous AI agents the ability to execute actions such as file editing, re-trieval, build, testing, and CLI commands directly within the repository to achieve user-defined objectives, thus enabling the completion of complex tasks autonomously. AutoDev offers the following key features: (i) the ability to track and manage user and AI agents conversations through a Conversation Manager, (ii) a library of customized Tools to accomplish a variety of code and SE related objectives, (iii) the ability to schedule various AI agents to work col-laboratively towards a common objective through an Agent Scheduler, and (iv) the ability to execute code and run tests through an Evaluation Environment. Conversation Manager Tools Agent Scheduler Evaluation Environment Figure 1 illustrates a high-level example of the AutoDev workflow. The user defines an objective (e.g., testing a specific method). The AI Agent writes tests in a new file and initiates the test execution command, all within a secure Evaluation Environment. The output of the test execution, including failure logs, is then incorporated into the conver-sation. The AI agent analyzes this output, triggers a retrieval command, incorporates the retrieved information by editing the file, and re-invokes the test execution. Finally, the environment provides feedback on the success of the test execution and completion of the user’s objective. 1 The entire process is orchestrated by AutoDev autonomously, requiring no developer intervention beyond setting the ini-tial objective. In contrast, with existing AI coding assistants integrated into IDEs, developers would have to manually execute tests (e.g., run pytest), provide failure logs to the AI chat interface, possibly identify additional contextual information to be incorporated, and repeat validation actions to ensure test success after the AI generates revised code. AutoDev draws inspiration from previous work in the area of autonomous AI agents. For example, AutoGen [22] is a framework which orchestrates language model workflows and facilitates conversations between multiple agents. AutoDev extends AutoGen by going beyond conversation management and enabling agents to directly interact with the code repository, executing commands and actions autonomously. Similarly, AutoDev builds upon Auto-GPT [8], an open-source AI agent for autonomous task execution by offering code and IDE specific capabilities to enable execu-tion of complex software engineering tasks. 22 8 In our evaluation, we assess the capabilities of AutoDev us-ing the HumanEval dataset[5], originally designed for code generation from natural language descriptions (docstrings). Additionally, we extend the evaluation to include the test case generation task, showcasing the versatility of AutoDev in handling diverse software engineering objectives. The results demonstrate promising performance, with AutoDev achieving remarkable scores of 91.5% and 87.8% for Pass@1 for code generation and test generation respectively. These outcomes underscore the effectiveness of AutoDev in automating software engineering tasks while maintaining a secure and user-controlled development environment. 5 2 AutoDev Design The design overview depicted in Fig. 2 provides insight into AutoDev’s architecture. Once the initial configurations are complete, AutoDev organizes its capabilities into four groups: a Conversation Manager that tracks and manages the user and agents conversations; a Tools library where a variety of code and IDE related tools are available for the agents, an Agents Scheduler that schedules various agents, and an Evaluation Environment that enables execution operations. Below we explain each capability in details. 2 2.1 Rules, Actions, and Objective Configuration The user initiates the process by configuring rules and actions through yaml files. These files define the available commands (actions) that AI agents can perform. Users can leverage default settings or fine-grained permissions by enabling/disabling specific commands, tailoring AutoDev to their specific needs. This configuration step allows for precise control over the AI agents’ capabilities. At this stage the user can define the number and behavior of the AI agents, assigning specific responsibilities, permissions, and available actions. For example, the user could define a "Developer" agent and a "Reviewer" agent, that collaboratively work towards an objective. Following the rules and actions configuration, the user specifies the software engineering task or process to be accomplished by AutoDev. For example, the user can ask to generate test cases and make sure they are syntactically correct, pass, and that do not contain bugs (this involves editing files, running test suite, executing syntax checks and bug-finding tools). 2.2 Conversation Manager The Conversation Manager, responsible for initialising the conversation history, plays a pivotal role in overseeing the high-level management of the ongoing conversation. It handles the task of deciding when to interrupt the process and ensures seamless communication between the user, AI agents, and the system as a whole. It maintains a conversation object which includes the messages from the AI agents and the result of the actions from the Evaluation Environment. 2.2.1 Parser. The Parser interprets the responses generated by agents, extracting commands and arguments in a predefined format. It ensures that the instructions are correctly formatted, validating the number and accuracy of arguments (e.g., a file editing command requires the file path argument). In case of parsing failures, error messages are injected into the conversation, preventing further actions on the repository. Successfully parsed commands are further analysed, by enforcing specific agent permissions and conducting additional semantic checks. It ensures that the suggested actions comply with the fine-grained permissions specified by the user. If the command passes scrutiny, the conversation manager invokes the corresponding action in the tools library. 2.2.1 Parser. 2.2.2 Output Organizer. The Output Organizer module processes the output received from the Evaluation Environment. It selects crucial information, such as status or errors, optionally summarizes relevant content, and adds a well-structured message to the conversation history. This ensures that the user has a clear and organized record of the Au-toDev’s actions and outcomes. 2.2.2 Output Organizer. Figure 2. Overview of the AutoDev Framework: The user initiates the process by defining the objective to be achieved. The Conversation Manager initializes the conversation and settings. The Agent Scheduler orchestrates AI agents to collaborate on the task and forwards their commands to the Conversation Manager. The Conversation Manager parses these commands and invokes the Tools Library, which offers various actions that can be performed on the repository. Agents’ actions are executed within a secure Docker environment, and the output is returned to the Conversation Manager, which incorporates it into the ongoing conversation. This iterative process continues until the task is successfully completed. Figure 2. 2.2.3 Conversation Conclusion. The Conversation Manager determines when to conclude the conversation. This may happen when an agent signals the completion of the task (stop command), the conversation reaches a user-defined maximum number of iterations/tokens, or issues are detected either in the process or within the Evaluation Environment. The comprehensive design of AutoDev ensures a systematic and controlled approach to AI-driven development. 2.2.3 Conversation Conclusion. 2.3 Agent Scheduler The Agent Scheduler takes charge of orchestrating AI agents to achieve the user-defined objective. Agents, configured with specific personas and sets of available commands, operate collaboratively to perform diverse tasks. The scheduler employs various collaboration algorithms, such as Round Robin, Token-Based, or Priority-Based, to determine the order and manner in which agents contribute to the conversation. Specifically, scheduling algorithms include but not limited to: (i) Round Robin collaboration, which invokes each agent sequentially, allowing each to execute a predeter-mined number of operations; (ii) token-based collaboration, enabling an agent to undertake multiple operations until it issues a token signifying completion of its assigned tasks; (iii) priority-based collaboration, initiating agents in accordance with their prioritized order. The agent scheduler invokes a specific agent passing the current conversation. 2.3.1 Agents. Agents, comprising Large Language Models (LLMs) like OpenAI GPT-4 and Small Language Models (SLMs) optimized for code generation, communicate through textual natural language. These agents receive objectives and conversation histories from the Agent Scheduler, responding with actions specified by the Rules and Actions configuration. Each agent, with its unique configuration, contributes to the overall progress toward achieving the user’s objective. 2.3.1 Agents. 2.4 Tools Library The Tools Library within AutoDev provides a range of commands that empower Agents to perform diverse operations on the repository. These commands are designed to encapsulate complex actions, tools, and utilities behind a straightforward and intuitive command structure. For instance, intricacies related to build and test execution are abstracted away through simple commands like build and test . • File Editing: This category encompasses commands for editing files, including code, configuration, and documentation. The utilities within this category, such as write, edit, insert, and delete, offer varying levels of granularity. Agents can perform actions ranging from writing entire files to modifying specific lines within a file. For example, the command write - , allows the agent to rewrite a range of lines with new content. File Editing - Retrieval: In this category, retrieval utilities range from basic CLI tools like grep, find, and ls to more sophisticated embedding-based techniques. These techniques enable Agents to look up similar code snippets, enhancing their ability to retrieve relevant information from the codebase. For example, the command retrieve allows the agent to perform embedding-based retrieval of similar snippets to the provided content. Build & Execution: Commands in this category allow Agents to compile, build, and execute the codebase effortlessly with simple and intuitive commands. The intricacies of low-level build commands are abstracted, streamlining the process within the Evaluation Environment infrastructure. Examples of commands in this category include: build, run . Testing & Validation: These commands enable Agents to test the codebase by executing a single test case, a specific test file, or the entire test suite. Agents can perform these actions without relying on low-level commands specific to particular testing frameworks. This category also encompasses validation tools such as linters and bug-finding utilities. Examples of commands in this category include: syntax which checks the syntax correctness, and test which runs the entire test suite. Git: Fine-grained permissions for git operations can be configured by the user. This includes operations like commits, push, and merges. For instance, Agents can be granted permission to perform only local commits or, if necessary, push changes to the origin repository. Communication: Agents can invoke a set of commands aimed at facilitating communication with other agents and/or the user. Notably, the talk command enables sending natural language messages (not interpreted as commands for repository actions), the ask command is used to request user feedback, and the stop command interrupts the process, indicating goal achievement or the inability of the agents to proceed further. Retrieval: In this category, retrieval utilities range from basic CLI tools like grep, find, and ls to more sophisticated embedding-based techniques. These techniques enable Agents to look up similar code snippets, enhancing their ability to retrieve relevant information from the codebase. For example, the command retrieve allows the agent to perform embedding-based retrieval of similar snippets to the provided content. Retrieval Build & Execution: Commands in this category allow Agents to compile, build, and execute the codebase effortlessly with simple and intuitive commands. The intricacies of low-level build commands are abstracted, streamlining the process within the Evaluation Environment infrastructure. Examples of commands in this category include: build, run . Build & Execution . Testing & Validation: These commands enable Agents to test the codebase by executing a single test case, a specific test file, or the entire test suite. Agents can perform these actions without relying on low-level commands specific to particular testing frameworks. This category also encompasses validation tools such as linters and bug-finding utilities. Examples of commands in this category include: syntax which checks the syntax correctness, and test which runs the entire test suite. Testing & Validation Git: Fine-grained permissions for git operations can be configured by the user. This includes operations like commits, push, and merges. For instance, Agents can be granted permission to perform only local commits or, if necessary, push changes to the origin repository. Git Communication: Agents can invoke a set of commands aimed at facilitating communication with other agents and/or the user. Notably, the talk command enables sending natural language messages (not interpreted as commands for repository actions), the ask command is used to request user feedback, and the stop command interrupts the process, indicating goal achievement or the inability of the agents to proceed further. Communication The Tools Library in AutoDev thus provides a versatile and accessible set of tools for AI Agents to interact with the codebase and communicate effectively within the collaborative development environment. 2.5 Evaluation Environment Running within a Docker container, the Evaluation Environment allows secure execution of file editing, retrieval, build, execution, and testing commands. It abstracts away the complexity of low-level commands, providing a simplified interface for agents. The Evaluation Environment returns standard output/error to the Output Organizer module. 2.6 Putting Everything Together The user initiates the conversation by specifying the objective and associated settings. The conversation manager initializes a conversation object, consolidating messages from both AI agents and the Evaluation Environment. Subsequently, the conversation manager dispatches the conversation to the Agent Scheduler, responsible for coordinating the actions of AI agents. In their role as AI agents, Language Models (Large or Small LMs) suggest commands through textual interactions. The Commands Interface encompasses a diverse set of functionalities, including File Editing, Retrieval, Build and Execution, Testing, and Git operations. These suggested commands are then parsed by the Conversation Manager, which subsequently directs them to the Evaluation Environment for execution on the codebase. Execution of these commands occurs within the secure confines of the Evaluation Environment, encapsulated within a Docker container. Post-execution, the resulting actions seamlessly integrate into the conversation history, contributing to subsequent iterations. This iterative process persists until the task is deemed complete by the Agents, user intervention occurs, or the maximum iteration limit is reached. AutoDev’s design ensures a systematic and secure orchestration of AI agents to achieve complex software engineering tasks in an autonomous and user-controlled manner. 3 Empirical Design In our empirical evaluation, we aim to assess AutoDev’s capabilities and effectiveness in software engineering tasks, examining whether it can enhance an AI model’s performance beyond simple inference. Additionally, we are interested in evaluating the cost of AutoDev in terms of the number of steps, inference calls, and tokens. We define three experimental research questions: Research Questions 1. 𝑅𝑄1: How effective is AutoDev in code generation task? 2. 𝑅𝑄2: How effective is AutoDev in test generation task? 3. 𝑅𝑄3: How efficient is AutoDev in completing tasks? 𝑅𝑄1: How effective is AutoDev in code generation task? To address 𝑅𝑄1, we evaluate AutoDev’s performance in a code generation task using the HumanEval problem-solving dataset in Python. This dataset comprises 164 handwritten programming problems, each containing a function signature, docstring, body, and an average of 7.7 unit tests. In our assessment, AutoDev is provided with a partial file containing the function signature and docstring, with the objective of implementing the method. We gauge AutoDev’s effectiveness using the Pass@k metric, where 𝑘 represents the number of attempts made. A successfully solved problem is defined as one where Au-toDev generates the code of the method’s body, satisfying all human-written tests. An attempt corresponds to an entire AutoDev conversation, which involves multiple inference calls and steps. This contrasts with other approaches, like directly invoking GPT-4, which typically involve a single inference call. Details regarding the multiple inference calls and steps are further explored in 𝑅𝑄3. For this evaluation, we set 𝑘 = 1, thus computing Pass@1, considering only the success achieved in the first attempt. 𝑅𝑄2: How effective is AutoDev in test generation task? For this research question, we modify the HumanEval dataset to evaluate AutoDev’s capabilities in test generation. We consider the human-written solution and discard the provided human-written tests. AutoDev is instructed to generate test cases for the focal method and is evaluated based on test success, invocation of the focal method, and test coverage. We report Pass@1, considering tests successful if they pass and invoke the focal method. Additionally, we compare the coverage of AutoDev’s tests with those written by humans. 𝑅𝑄3: How efficient is AutoDev in completing tasks? In this research question, we investigate AutoDev’s efficiency in completing SE tasks. We analyze the number of steps or inference calls needed, the distribution of commands used (e.g., write, test), and the total number of tokens used in the conversation. AutoDev Settings For this evaluation, AutoDev maintains consistent settings with one agent based on the GPT-4 model (gpt-4-1106-preview). Enabled actions include file editing, retrieval, and testing. The only communication command available is the stop command, indicating task completion. Other commands, such as ask, are disabled, requiring AutoDev to operate autonomously without human feedback or intervention beyond the initial goal-setting. 4 Empirical Results 𝑅𝑄1: How effective is AutoDev in a code generation task? Table 1 displays the results for 𝑅𝑄1, comparing AutoDev against two alternative approaches and the zero-shot baseline. The table includes information about the model powering each approach, the need for additional training, and the Pass@1 metric. 1 We compared AutoDev with Language Agent Tree Search (LATS) and Reflexion, two leading approaches on the Hu-manEval leaderboard as of March 2024 [1]. The results for the zero-shot baseline (GPT-4) are taken from the OpenAI GPT-4 technical report [11], while those for LATS and Reflexion from the HuamnEval leaderboard [1]. 1 11 1 Language Agent Tree Search (LATS) [23] is a versatile framework that utilizes Large Language Models (LLMs) for planning, acting, and reasoning. Inspired by Monte Carlo tree search, LATS employs LLMs as agents, value functions, and optimizers, repurposing their capabilities for improved decision-making. 23 Reflexion[17] introduces a unique framework to reinforce language agents via linguistic feedback without weight updates. Agents in Reflexion verbally reflect on task feedback signals, maintaining their reflective text in an episodic memory buffer for enhanced decision-making. 17 Table 1 indicates that AutoDev achieves a Pass@1 rate of 91.5%, securing the second-best position on the HumanEval leaderboard. Notably, this result is obtained without addi-tional training data, distinguishing AutoDev from LATS, which achieves 94.4%. Furthermore, the AutoDev framework enhances GPT-4 performance from 67% to 91.5%, marking a 30% relative improvement. 1 These results underscore AutoDev’s capability to significantly enhance the performance of LLMs in completing software engineering tasks. However, it’s worth noting that AutoDev’s effectiveness may entail multiple inference calls and steps, as we delve into further detail in 𝑅𝑄3. 𝑅𝑄2: How effective is AutoDev in test generation task? Table 2 presents the outcomes for 𝑅𝑄2, comparing AutoDev with the zero-shot GPT-4 (baseline) and human-written tests in the test generation task. Since OpenAI has not evaluated GPT-4 on the test generation task, we obtained the zero-shot GPT-4 results by invoking the inference with instruction prompting on the same GPT-4 model used for AutoDev. 2 AutoDev attains a Pass@1 score of 87.8% on the HumanEval dataset modified for the test generation task, exhibiting a 17% relative improvement over the baseline that utilizes the same GPT-4 model. The correct tests generated by AutoDev (included in Pass@1) achieve a robust 99.3% coverage, comparable to the human-written tests’ coverage of 99.4%. Additionally, Table 2 reports the overall coverage across the entire dataset of test cases, considering incorrect or failing tests as lacking coverage. In this regard, AutoDev achieves an 88.8% coverage over the complete dataset of focal methods. 2 These results affirm AutoDev’s prowess in addressing diverse software engineering tasks. 𝑅𝑄3: How efficient is AutoDev in completing tasks? Figure 3 illustrates the cumulative number of commands used by AutoDev for both the Code Generation and Test Generation tasks, considering the average number of commands used for the evaluation of each HumanEval problem in 𝑅𝑄1 and 𝑅𝑄2. 3 For Code Generation, AutoDev executed an average of 5.5 commands, comprising 1.8 write operations, 1.7 test operations, 0.92 stop operations (indicating task completion), 0.25 incorrect commands, along with minimal retrieval (grep, find, cat), syntax check operations, and talk communication commands. In the case of Test Generation, the average number of commands aligns with the Code Generation task. However, Test Generation involves more retrieval operations and an increased occurrence of incorrect operations, resulting in a total average of 6.5 commands for each run. We classify a command as incorrect if it references an unavailable command or fails parsing (e.g., incorrect formatting or parameter count). The most prevalent incorrect commands involve AI agents mixing natural language with code or commands. Such issues could potentially be addressed through more flexible parsing or improved prompting, as discussed further in the Discussion section. While AutoDev incurs more inference calls compared to approaches that generate candidate code in a single call, it is essential to note that AutoDev also performs testing and validation operations, tasks typically executed by developers to validate generated candidates. Testing and syntax operations invoked by AutoDev would be undertaken regardless by developers receiving code generated by AI, such as Copilot. Furthermore, AutoDev often communicates task completion through talk commands, providing insights and interpretability of the solution. Another notable communication command contributing to the overall count is the stop command. This represents a relatively inexpensive inference call, generating only one token. Potential optimizations could involve batching such operations with other commands or inferences. The average length of AutoDev conversations to solve each HumanEval problem in 𝑅𝑄1 and 𝑅𝑄2 is 1656 and 1863 tokens respectively. This encompasses the user’s goal, messages from the AI agent, and responses from the Evaluation Environment. In comparison, the zero-shot GPT-4 (baseline) uses 200 tokens (estimated) for code generation and 373 tokens for test generation, on average, for each task. While AutoDev uses more tokens, a significant amount are spent for testing, validation, and explanation of its own generated code, going beyond what the baseline approach offers. Finally, AutoDev incurs execution costs related to orchestrating AI Agents, managing conversations, and executing commands within a Docker environment. Notably, the Docker-based evaluation environment accounts for the major execution cost, exhibiting higher overhead compared to direct CLI commands within the user’s environment. However, we designed AutoDev with security as a top priority, ensuring the secure execution and validation of AI-generated code. 5 Discussion 5.1 AutoDev in Action Figures 4 (Part I) and 5 (Part II) provide a glimpse of Au-toDev in action during a test generation task, with some interactions summarized or omitted for brevity. Initiating from Figure 4, we set the goal for AutoDev to generate pytest test cases following a specific format. The AutoDev agent initiates the write-new command, providing the filepath and content of the test file. AutoDev executes the operation, confirming successful content writing to the agent. Subsequently, the AutoDev agent triggers the test operation, and AutoDev runs the test within its secure Docker environment, presenting the test execution report JSON (summarized in the figure for brevity). 4 5 4 The AutoDev agent identifies an error in the pytest output, recognizing that a fix is required to align the test with the expected behavior of the function. Continuing in Figure 5, the AutoDev agent issues the write command, specifying the filepath and line number range (5-5) for rewriting the incorrect assert statement. Following this, the AutoDev agent proceeds with the test execution, which now succeeds. Recognizing the completion of the goal, the AutoDev agent concludes the conversation. 5 This example highlights AutoDev’s capability to self-evaluate its generated code and address bugs within its own output. Additionally, it demonstrates how AutoDev facilitates user insight into agent actions, allowing agents to communicate during the task. 5.2 Multi-Agent Collaboration AutoDev supports multi-agent collaboration on tasks, orchestrated by the Agent Scheduler. In our evaluation, given the relative simplicity of the HumanEval dataset, we limited our setup to a single GPT-4 agent for the tasks. However, preliminary results indicate the positive impact of multi-agent collaboration on more complex tasks. Experimenting with an AI Developer and AI Reviewer, each with distinct responsibilities and available actions, collaborating on fixing a complex bug revealed interesting interactions. The AI Reviewer could pre-emptively identify AI Developer mistakes before code validation actions were executed and provide relevant suggestions. Our future plans involve expanding evaluations to incorporate more complex scenarios where multi-agent collaboration can significantly enhance AutoDev’s performance. 5.3 Human in the Loop AutoDev allows AI agents to communicate progress on tasks or request human feedback using the talk and ask commands, respectively. Anecdotally, these commands have proven helpful for developers using AutoDev to understand the agent’s intentions and gain insights into the agent’s plan. The addition of the ask command was in direct response to a developer’s request during our pilot study, where they wanted the ability to provide feedback when the agents appeared uncertain about the next actions. Our future plans involve deeper integration of humans within the AutoDev loop, allowing users to interrupt agents and provide prompt feedback. 5.4 AutoDev Integrations Our pilot study involved developers using AutoDev as a CLI command, with the conversation available for observation within the VSCode IDE. Moving forward, our goal is to integrate AutoDev into IDEs, creating a chatbot experience, and incorporate it into CI/CD pipelines and PR review platforms. We envision developers assigning tasks and issues to AutoDev, reviewing results within a PR system, and further streamlining the software development workflow. 6 Related Work Our work builds upon an extensive body of literature that applies AI to various Software Engineering tasks. In this section, we explore recent developments and contextualize AutoDev within this rich research landscape. 6.1 AI in Software Engineering The integration of AI, particularly Large Language Models (LLMs), into software engineering has witnessed substantial progress. Models like GPT-3 [7], InstructGPT [14], and GPT-4 [13] have leveraged the Transformer architecture [20] to comprehend and generate not only natural language, but also source code. The massive parameter sizes of LLMs, such as those in Gropher [15] and Megatron-Turing NLG [18], and GPT-4 [13] allowed these AI models to achieve impressive performance across diverse tasks. 7 14 13 20 15 18 13 As software development practices continue to evolve, the integration of cutting-edge technologies becomes paramount for enhancing developer productivity [5]. Among the notable advancements, the utilization of LLMs within Integrated Development Environments (IDEs) has garnered significant attention [4, 10]. LLMs, including prominent models such as OpenAI’s GPT-3.5 [12] and GPT-4 [13], as well as robust open-source models like Code Llama [16], exhibit the potential to act as intelligent programming assistants. 5 4 10 12 13 16 In this paper, we introduce AutoDev, a comprehensive framework for autonomous software engineering tasks within a secure development environment. AutoDev extends beyond existing works by providing a versatile tools library, empowering AI agents to autonomously perform intricate tasks, such as code editing, testing, and integration. AutoDev is also LLM-agnostic, with an infrastructure that allows a diverse set of AI models, with different parameter size and architectures, to collaborate on a given task. 6.2 Evaluation of LLMs in Software Engineering Evaluating LLMs for software engineering tasks poses unique challenges. Traditional language-based metrics, such as BLEU, have been the focus of previous research, with evaluations conducted on static datasets like GLUE [21] and BIGBench [19]. However, these metrics often fall short in capturing essential programming aspects like syntax correctness and execution-based metrics such as build and testing. 21 19 CodeXGLUE [9] has addressed these limitations by providing a comprehensive evaluation platform for LLMs in software engineering. It offers a diverse benchmark dataset along with baseline models like CodeBERT and CodeGPT. 9 HumanEval [5] contributes to the field by focusing on the functional correctness of LLMs, introducing a benchmark dataset of hand-written programming problems in Python. In recent developments, the Copilot Evaluation Harness 5 [3] builds upon prior works in the literature, aiming to enhance their contributions. Similar to HumanEval, the Copilot Evaluation Harness incorporates considerations of code execution, but extending the spectrum of software engineering tasks (code, test, and documentation generation, workspace understanding and query resolution) as well as increasing the metrics used for evaluation. This evaluation harness also encompasses large and real-world codebases. 3 While our current evaluation relies on HumanEval for assessing the effectiveness of AutoDev in two coding tasks, our future work aims to extend this assessment to more challenging and real-world datasets, such as those offered by the Copilot Evaluation Harness. 6.3 AI in Software Engineering Interactions While prior works have explored the intersection of AI and software engineering, few have delved into AI-guided programming within IDE interactions. AutoDev, as introduced in this paper, draws inspiration from existing works in the literature while enhancing their contributions. Notable examples include Auto-GPT[8], LATS (Language Agent Tree Search)[23], and Reflexion[17], each presenting unique approaches to AI-driven tasks. 8 23 17 Auto-GPT[8] operates by pairing GPT-3.5 and GPT-4 with a companion bot, enabling users to instruct these language models on specific goals. The companion bot utilizes GPT-3.5 and GPT-4, along with various programs, to execute the necessary steps for goal achievement. 8 LATS [23] , on the other hand, is a general framework that synergizes the capabilities of LLMs in planning, acting, and reasoning. Inspired by Monte Carlo tree search, commonly used in model-based reinforcement learning, LATS employs LLMs as agents, value functions, and optimizers, enhancing decision-making. It introduces an environment for external feedback, offering a deliberate and adaptive problem-solving mechanism. 23 Reflexion[17] introduces a novel framework to reinforce language agents through linguistic feedback. Reflexion agents verbally reflect on task feedback signals, maintaining their reflective text in an episodic memory buffer for improved decision-making. This flexible framework incorporates various types and sources of feedback signals and exhibits significant improvements over baseline agents in diverse tasks, including sequential decision-making, coding, and language reasoning. 17 AutoDev specializes these ideas for the Software Engineering realm, offering a flexible framework that allows AI agents to complete complex SE tasks in full autonomy. Our work aims to bridge the gap between traditional software engineering practices and AI-driven automation, facilitating collaborative efforts between developers and AI agents. By introducing a versatile tools library, AutoDev empowers AI agents to autonomously perform intricate tasks, providing a promising advancement in the landscape of AI-assisted software development. 7 Conclusion In this paper, we introduced AutoDev, a framework enabling AI Agents to autonomously interact with repositories, perform actions, and tackle complex software engineering tasks. We’ve shifted the responsibility of extracting relevant context for software engineering tasks and validating AI-generated code from users (mainly developers) to the AI agents themselves. Agents are now empowered to retrieve context through Retrieval actions and validate their code generation through Build, Execution, Testing, and Validation actions. The developer’s role within the AutoDev framework transforms from manual actions and validation of AI suggestions to a supervisor overseeing multi-agent collaboration on tasks, with the option to provide feedback. Developers can monitor AutoDev’s progress toward goals by observing the ongoing conversation used for communication among agents and the repository. Our evaluation on the HumanEval dataset for code and test generation showcased impressive results, achieving a Pass@1 score of 91.5 for code generation—a second-best result on the leaderboard at the time of writing, and the best among approaches requiring no extra training data. AutoDev also excelled in test generation with a Pass@1 score of 87.8%, achieving a 99.3% coverage from passing tests. Looking ahead, our goal for future work is to integrate AutoDev into IDEs as a chatbot experience and incorporate it into CI/CD pipelines and PR review platforms. References [1] Code generation on humaneval - state-of-the-art. hNps:// paperswithcode.com/sota/code-generation-on-humaneval, 2024. Ac-cessed: 2024-02-27. hNps:// paperswithcode.com/sota/code-generation-on-humaneval, [2] Github copilot: Your ai pair programmer. hNps://github.com/features/ copilot, 2024. hNps://github.com/features/ copilot, [3] Agarwal, A., Chan, A., Chandel, S., Jang, J., Miller, S., Moghad-dam, R. Z., Mohylevskyy, Y., Sundaresan, N., and Tufano, M. Copi-lot evaluation harness: Evaluating llm-guided software programming, 2024. [4] Chen, B., Mustakin, N., Hoang, A., Fuad, S., and Wong, D. Vscuda: Llm based cuda extension for visual studio code. In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis (New York, NY, USA, 2023), SC-W ’23, Association for Computing Machinery, p. 11–17. Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis [5] Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., et al. Evaluating large language models trained on code. [6] Ding, Y., Wang, Z., Ahmad, W., Ding, H., Tan, M., Jain, N., Ra-manathan, M. K., Nallapati, R., Bhatia, P., Roth, D., et al. Cross-codeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems 36 (2024). Advances in Neural Information Processing Systems 36 [7] Floridi, L., and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30 (2020), 681–694. Minds and Machines 30 [8] Gravitas, S. Autogpt. hNps://github.com/Significant-Gravitas/ AutoGPT, 2024. GitHub repository. hNps://github.com/Significant-Gravitas/ AutoGPT, [9] Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning benchmark dataset for code understanding and generation, 2021. [10] Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., and Myers, B. In-ide generation-based information support with a large language model, 2023. [11] OpenAI, :, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brak-man, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cum-mings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Łukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Mal-facini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024. [12] OpenAI. Gpt 3.5 models, 2023. [13] OpenAI. Gpt-4 technical report, 2023. [14] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training lan-guage models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744. Advances in Neural Information Processing Systems 35 [15] Rae, J. W., Borgeaud, S., Cai, T., Millican, K., and Others. Scaling language models: Methods, analysis & insights from training gopher, 2022. [16] Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). arXiv preprint arXiv:2308.12950 [17] Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning, 2023. [18] Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., Zhang, E., Child, R., Aminabadi, R. Y., Bernauer, J., Song, X., Shoeybi, M., He, Y., Houston, M., Tiwary, S., and Catanzaro, B. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, 2022. [19] Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. [20] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems 30 (2017). Advances in neural information processing systems 30 [21] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. [22] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023. [23] Zhou, A., Yan, K., Shlapentokh-Rothman, M., Wang, H., and Wang, Y.-X. Language agent tree search unifies reasoning acting and planning in language models, 2023. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv