Rethinking Pair Programming for the AI Era

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Contexts, Methods, and Tasks 2. Contexts, Methods, and Tasks 3. Mixed Outcomes 3. Mixed Outcomes 3.1. Quality and 3.2 Productivity 3.1. Quality and 3.2 Productivity 3.3. Learning and 3.4 Cost 3.3. Learning and 3.4 Cost 4. Moderators 4. Moderators 4.1. Task Types & Complexity 4.1. Task Types & Complexity 4.2. Compatibility 4.2. Compatibility 4.3. Communication 4.3. Communication 4.4. Collaboration 4.4. Collaboration 4.5. Logistics 4.5. Logistics 5. Discussion and Future Work 5.1. LLM, Your pAIr Programmer? 5.1. LLM, Your pAIr Programmer? 5.2. LLM, A Better pAIr Programmer? 5.2. LLM, A Better pAIr Programmer? 5.3. LLM, Students’ pAIr Programmer? 5.3. LLM, Students’ pAIr Programmer? 6. Conclusion, Acknowledgments, and References 6. Conclusion, Acknowledgments, and References 6 CONCLUSION This paper has discussed the concept of human-AI pair programming (pAIr programming). We found that both human-human and human-AI pair programming have benefits and challenges, but current research did not give us a clear answer on the efficacy of human-AI pair programming. Human-human pair programming literature yield insights on what study designs should pAIr researchers adopt (e.g., more realistic observations), what outcomes and measures should pAIr researchers use to evaluate their work (e.g., use more valid quality and productivity measurements, and further investigate cost), and what moderators should pAIr researchers consider to further analyze the pAIr process and improve pAIr design (e.g, compatibility, communication, etc.). In conclusion, more valid and comprehensive measurements are needed to evaluate pAIr, more comparisons can be drawn between human-human vs. human-AI pair programming, and more works can explore how to best support LLM-assisted programming with insights from the rich literature on human-human pair programming. ACKNOWLEDGMENTS Thanks to Ken’s lab members for giving feedback on this work. Thanks to Dr. Stephen MacNeil for coming up with the creative “pAIr” keyword for this project. REFERENCES [1] Gati Aher, Rosa I Arriaga, and Adam Tauman Kalai. 2022. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. (Aug. 2022). arXiv:2208.10264 [cs.CL] http: //arxiv.org/abs/2208.10264 [2] Wolfgang Ahrendt, Richard Bubel, and Reiner Hähnle. 2009. Integrated and Tool-Supported Teaching of Testing, Debugging, and Verification. In Teaching Formal Methods. Springer Berlin Heidelberg, 125–143. https://doi.org/10.1007/978-3-642-04912-5_9 [3] Naser Al Madi. 2023. How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/3551349.3560438 [4] Mustafa Ally, Fiona Darroch, and Mark Toleman. 2005. A framework for understanding the factors influencing pair programming success. In Extreme Programming and Agile Processes in Software Engineering. Springer Berlin Heidelberg, Berlin, Heidelberg, 82–91. https://doi. org/10.1007/11499053_10 [5] Carolina Alves De Lima Salge and Nicholas Berente. 2016. Pair Programming vs. Solo Programming: What Do We Know After 15 Years of Research?. In 2016 49th Hawaii International Conference on System Sciences (HICSS). 5398–5406. https://doi.org/10.1109/HICSS.2016.667 [6] Erik Arisholm, Hans Gallis, Tore Dyba, and Dag I K Sjoberg. 2007. Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise. IEEE Trans. Software Eng. 33, 2 (Feb. 2007), 65–86. https://doi.org/10.1109/TSE.2007.17 [7] Owura Asare, Meiyappan Nagappan, and N Asokan. 2022. Is GitHub’s Copilot as Bad as Humans at Introducing Vulnerabilities in Code? (April 2022). arXiv:2204.04741 [cs.SE] http://arxiv.org/abs/2204.04741 [8] Shraddha Barke, Michael B James, and Nadia Polikarpova. 2022. Grounded Copilot: How Programmers Interact with Code-Generating Models. (June 2022). arXiv:2206.15000 [cs.HC] http://arxiv.org/abs/ 2206.15000 [9] Kent Beck. 1999. Extreme programming explained: embrace change. Addison-Wesley Longman Publishing Co., Inc., USA. https://dl.acm. org/doi/10.5555/318762 [10] Brett A Becker, Paul Denny, James Finnie-Ansley, Andrew LuxtonReilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 500–506. https://doi.org/10.1145/3545945.3569759 [11] Andrew Begel and Nachiappan Nagappan. 2008. Pair programming: what’s in it for me?. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement (Kaiserslautern Germany). ACM, New York, NY, USA. https: //doi.org/10.1145/1414004.1414026 [12] Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2023. Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools. Queueing Syst. 20, 6 (Jan. 2023), 35–57. https: //doi.org/10.1145/3582083 [13] Bobby Bodenheimer, B Sanders, M R Kramer, K Viswanath, R Balachandran, Kadira Belynne, and Gautam Biswas. 2009. Construction and Evaluation of Animated Teachable Agents. J. Educ. Technol. Soc. (2009). https://www.semanticscholar.org/paper/ 2899ac4dfe209db4767ec01b5df337079bada517 [14] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. (March 2023). arXiv:2303.12712 [cs.CL] http://arxiv.org/abs/2303.12712 [15] Chen Cao. 2023. Scaffolding CS1 Courses with a Large Language Model-Powered Intelligent Tutoring System. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23 Companion). Association for Computing Machinery, New York, NY, USA, 229–232. https://doi.org/10. 1145/3581754.3584111 [16] E A Chaparro, Aybala Yuksel, Pablo Romero, and Sallyann Bryant. 2005. Factors Affecting the Perceived Effectiveness of Pair Programming in Higher Education. Annual Workshop of the Psychology of Programming Interest Group (2005). https://www.semanticscholar. org/paper/c095f0d9b17cd9c2851000534740e7cc087253fa [17] Jan Chong and Tom Hurlbutt. 2007. The Social Dynamics of Pair Programming. In 29th International Conference on Software Engineering (ICSE’07). ieeexplore.ieee.org, 354–363. https://doi.org/10.1109/ICSE. 2007.87 [18] Alistair Cockburn and L Williams. 2001. The costs and benefits of pair programming. Computer Science (2001). https://www.semanticscholar. org/paper/5ff7b75b20fdbfae23587b660b7093aec2f48e69 [19] Bernardo José da Silva Estácio and Rafael Prikladnicki. 2015. Distributed Pair Programming: A Systematic Literature Review. Information and Software Technology 63 (July 2015), 1–10. https: //doi.org/10.1016/j.infsof.2015.02.011 [20] Wei Dai, Jionghao Lin, Flora Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gasevic, and Guanliang Chen. 2023. Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT. (April 2023). https://doi.org/10.35542/osf.io/hcgzj [21] Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, Zhen Ming, and Jiang. 2022. GitHub Copilot AI pair programmer: Asset or Liability? ArXiv (2022). https: //doi.org/10.48550/ARXIV.2206.15331 [22] James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew LuxtonReilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). Association for Computing Machinery, New York, NY, USA, 10–19. https://doi.org/10.1145/3511861.3511863 [23] Sue Fitzgerald, Renée McCauley, Brian Hanks, Laurie Murphy, Beth Simon, and Carol Zander. 2010. Debugging From the Student Perspective. IEEE Trans. Educ. 53, 3 (Aug. 2010), 390–396. https: //doi.org/10.1109/TE.2009.2025266 [24] S Freudenberg, Pablo Romero, and Benedict Du Boulay. 2007. Talking the talk: Is intermediate-level conversation the key to the pair programming success story?. In AGILE 2007. unknown, 84–91. https: //doi.org/10.1109/AGILE.2007.1 [25] Github. [n. d.]. GitHub Copilot Labs. https://githubnext.com/projects/ copilot-labs/. https://githubnext.com/projects/copilot-labs/ Accessed: 2023-5-19. [26] GitHub. 2021. Your AI pair programmer: Copilot. https://github. com/features/copilot. https://github.com/features/copilot Accessed: 2022-10-5. [27] Google. [n. d.]. Bard. https://bard.google.com/. https://bard.google. com/ Accessed: 2023-5-19. [28] Keun-Woo Han, Eunkyoung Lee, and Youngjun Lee. 2010. The Impact of a Peer-Learning Agent Based on Pair Programming in a Programming Course. IEEE Trans. Educ. 53, 2 (May 2010), 318–327. https://doi.org/10.1109/TE.2009.2019121 [29] Brian Hanks, Sue Fitzgerald, Renée McCauley, Laurie Murphy, and Carol Zander. 2011. Pair programming in education: a literature review. Comput. Sci. Educ. 21, 2 (June 2011), 135–173. https://doi.org/10.1080/ 08993408.2011.579808 [30] Jo E Hannay, Erik Arisholm, Harald Engvik, and Dag I K Sjoberg. 2010. Effects of Personality on Pair Programming. IEEE Trans. Software Eng. 36, 1 (Jan. 2010), 61–80. https://doi.org/10.1109/TSE.2009.41 [31] Jo E Hannay, Tore Dybå, Erik Arisholm, and Dag I K Sjøberg. 2009. The effectiveness of pair programming: A meta-analysis. Information and Software Technology 51, 7 (July 2009), 1110–1122. https://doi.org/ 10.1016/j.infsof.2009.02.001 [32] Steffi Heidig and Geraldine Clarebout. 2011. Do pedagogical agents make a difference to student motivation and learning? Educational Research Review 6, 1 (Jan. 2011), 27–54. https://doi.org/10.1016/j. edurev.2010.07.004 [33] Kenneth Holstein, Vincent Aleven, and Nikol Rummel. 2020. A Conceptual Framework for Human–AI Hybrid Adaptivity in Education. Artificial Intelligence in Education 12163 (June 2020), 240. https://doi.org/10.1007/978-3-030-52237-7_20 [34] John J Horton. 2023. Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? (Jan. 2023). arXiv:2301.07543 [econ.GN] http://arxiv.org/abs/2301.07543 [35] Saki Imai. 2022. Is GitHub Copilot a Substitute for Human Pairprogramming? An Empirical Study. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). ieeexplore.ieee.org, 319–321. https://doi.org/10. 1145/3510454.3522684 [36] Randall W Jensen. 2005. A Pair Programming Experience. ACCU - professionalism in programming Overload 13, 65 (Feb. 2005). https: //accu.org/journals/overload/13/65/jensen_254/ [37] Eirini Kalliamvakou. 2022. Research: quantifying GitHub Copilot’s impact on developer productivity and happiness. https://github.blog/2022-09-07-research-quantifying-githubcopilots-impact-on-developer-productivity-and-happiness/. https://github.blog/2022-09-07-research-quantifying-githubcopilots-impact-on-developer-productivity-and-happiness/ Accessed: 2022-10-13. [38] Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2022. Large Language Models are few-shot testers: Exploring LLM-based general bug reproduction. ArXiv (2022). https://doi.org/10.48550/ARXIV.2209.11515 [39] Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. (Feb. 2023). arXiv:2302.07427 [cs.HC] http://arxiv.org/ abs/2302.07427 [40] Shobhan Kumar, Arun Chauhan, and Pavan Kumar C. 2022. Learning Enhancement Using Question-Answer Generation for e-Book Using Contrastive Fine-Tuned T5. In Big Data Analytics. Springer Nature Switzerland, 68–87. https://doi.org/10.1007/978-3-031-24094-2_5 [41] Sandeep Kaur Kuttal, Bali Ong, Kate Kwasny, and Peter Robe. 2021. Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21, Article 243). Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/ 3411764.3445659 [42] Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. (April 2023). arXiv:2304.03938 [cs.CY] http://arxiv.org/abs/2304.03938 [43] Mengjun Li and Ayoung Suh. 2021. Machinelike or Humanlike? A Literature Review of Anthropomorphism in AI-Enabled Technology. In Hawaii International Conference on System Sciences 2021 (HICSS-54). https://aisel.aisnet.org/hicss-54/in/ai_based_assistants/5/ [44] Lijia Lin, Robert K Atkinson, Robert M Christopherson, Stacey S Joseph, and Caroline J Harrison. 2013. Animated agents and learning: Does the type of verbal feedback they provide matter? Comput. Educ. 67 (Sept. 2013), 239–249. https://doi.org/10.1016/j.compedu.2013.04. 017 [45] Kim Man Lui and Keith C C Chan. 2006. Pair programming productivity: Novice–novice vs. expert–expert. Int. J. Hum. Comput. Stud. 64, 9 (Sept. 2006), 915–925. https://doi.org/10.1016/j.ijhcs.2006.04.010 [46] Mary Margaret Lusk and Robert K Atkinson. 2007. Animated pedagogical agents: does their degree of embodiment impact learning from static or animated worked examples? Appl. Cogn. Psychol. 21, 6 (Sept. 2007), 747–764. https://doi.org/10.1002/acp.1347 [47] Phil Maguire, Rebecca Maguire, Philip Hyland, and Patrick Marshall. 2014. Enhancing collaborative learning using pair programming: Who benefits? AISHE-J 6, 2 (June 2014). https://ojs.aishe.org/index.php/ aishe-j/article/view/141 [48] Richard E Mayer. 2014. Principles based on social cues in multimedia learning: Personalization, voice, image, and embodiment principles. The Cambridge handbook of multimedia learning 16 (2014), 345–370. https://books.google.com/books?hl=en&lr= &id=r3rsAwAAQBAJ&oi=fnd&pg=PA345&ots=iUhQ53T8QY&sig= 5tQyKi_f-7aILMxRLuwGTLIix3c [49] Richard E Mayer and C Scott DaPra. 2012. An embodiment effect in computer-based learning with animated pedagogical agents. J. Exp. Psychol. Appl. 18, 3 (Sept. 2012), 239–252. https://doi.org/10.1037/ a0028616 [50] Renee McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: A Review of the Literature from an Educational Perspective. Computer Science Education 18, 2 (June 2008), 67–92. https://doi.org/10.1080/ 08993400802114581 [51] Charlie McDowell, Linda Werner, Heather Bullock, and Julian Fernald. 2002. The effects of pair-programming on performance in an introductory programming course. In Proceedings of the 33rd SIGCSE technical symposium on Computer science education (Cincinnati, Kentucky) (SIGCSE ’02). Association for Computing Machinery, New York, NY, USA, 38–42. https://doi.org/10.1145/563340.563353 [52] Charlie McDowell, Linda Werner, Heather E Bullock, and Julian Fernald. 2006. Pair programming improves student retention, confidence, and program quality. Commun. ACM 49, 8 (Aug. 2006), 90–95. https://doi.org/10.1145/1145287.1145293 [53] Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. 2022. Reading between the lines: Modeling user behavior and costs in AI-assisted programming. ArXiv (2022). https://doi.org/10.48550/ ARXIV.2210.14306 [54] Ambar Murillo and Sarah D’Angelo. 2023. An Engineering Perspective on Writing Assistants for Productivity and Creative Code. The Second Workshop on Intelligent and Interactive Writing Assistants (2023). https://cdn.glitch.global/d058c114-3406-43be-8a3cd3afff35eda2/paper1_2023.pdf [55] Laurie Murphy, Sue Fitzgerald, Brian Hanks, and Renée McCauley. 2010. Pair debugging: a transactive discourse analysis. In Proceedings of the Sixth international workshop on Computing education research (Aarhus, Denmark) (ICER ’10). Association for Computing Machinery, New York, NY, USA, 51–58. https://doi.org/10.1145/1839594.1839604 [56] Brad A Myers, Amy J Ko, Thomas D LaToza, and Youngseok Yoon. 2016. Programmers are users too: Human-centered methods for improving programming tools. Computer 49, 7 (July 2016), 44–52. https://doi. org/10.1109/MC.2016.200 [57] Nachiappan Nagappan, Laurie Williams, Miriam Ferzli, Eric Wiebe, Kai Yang, Carol Miller, and Suzanne Balik. 2003. Improving the CS1 experience with pair programming. In Proceedings of the 34th SIGCSE technical symposium on Computer science education (Reno Navada USA). ACM, New York, NY, USA. https://doi.org/10.1145/611892. 612006 [58] N Nguyen and Sarah Nadi. 2022. An Empirical Evaluation of GitHub Copilot’s Code Suggestions. 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR) (2022). https://doi.org/10. 1145/3524842.3528470 [59] Amy Ogan, Samantha Finkelstein, Elijah Mayfield, Claudia D’Adamo, Noboru Matsuda, and Justine Cassell. 2012. “Oh dear stacy!”: social interaction, elaboration, and learning with teachable agents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi.org/10.1145/2207676.2207684 [60] Venkata Vinod Kumar Padmanabhuni, Hari Praveen Tadiparthi, and Sagar Madina Muralidhar Yanamadala. 2012. Effective pair programming practice-an experimental study. Journal of Emerging Trends in Computing and Information Sciences 3, 4 (2012), 471–479. http://www.agilemethod.csie.ncu.edu.tw/agileMethod/ download/2012papers/2012%20Effective%20Pair%20Programming% 20Practice-%20An%20Experimental%20Study/Effective%20Pair% 20Programming%20Practice-%20An%20Experimental%20Study.pdf [61] Zachary A Pardos and Shreya Bhandari. 2023. Learning gain differences between ChatGPT and human tutor generated algebra hints. (Feb. 2023). arXiv:2302.06871 [cs.CY] http://arxiv.org/abs/2302.06871 [62] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan DolanGavitt, and Ramesh Karri. 2021. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. (Aug. 2021). arXiv:2108.09293 [cs.CR] http://arxiv.org/abs/2108.09293 [63] Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. (Feb. 2023). arXiv:2302.06590 [cs.SE] http://arxiv.org/abs/ 2302.06590 [64] Michael Perscheid, Benjamin Siegmund, Marcel Taeumel, and Robert Hirschfeld. 2017. Studying the advancement in debugging practice of professional software developers. Software Quality Journal 25, 1 (March 2017), 83–110. https://doi.org/10.1007/s11219-015-9294-2 [65] Laura Plonka, Judith Segal, Helen Sharp, and Janet van der Linden. 2011. Collaboration in Pair Programming: Driving and Switching. In Agile Processes in Software Engineering and Extreme Programming - 12th International Conference, XP 2011, Madrid, Spain, May 10-13, 2011. Proceedings, Vol. 77. unknown, 43–59. https://doi.org/10.1007/978-3- 642-20677-1_4 [66] James Prather, Brent N Reeves, Paul Denny, Brett A Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. (April 2023). arXiv:2304.02491 [cs.HC] http://arxiv.org/abs/ 2304.02491 [67] David Preston. 2006. Using collaborative learning research to enhance pair programming pedagogy. SIGITE Newsl. 3, 1 (Jan. 2006), 16–21. https://doi.org/10.1145/1113378.1113381 [68] Ben Puryear and Gina Sprint. 2022. Github copilot in the classroom: learning to code with AI assistance. J. Comput. Sci. Coll. 38, 1 (Dec. 2022), 37–47. https://dl.acm.org/doi/pdf/10.5555/3575618.3575622 [69] Peter Robe and Sandeep Kaur Kuttal. 2022. Designing PairBuddy—A Conversational Agent for Pair Programming. ACM Trans. Comput.- Hum. Interact. 29, 4 (May 2022), 1–44. https://doi.org/10.1145/3498326 [70] Norsaremah Salleh, Emilia Mendes, and John Grundy. 2011. Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review. IEEE Trans. Software Eng. 37, 4 (July 2011), 509–525. https://doi.org/10.1109/TSE.2010.59 [71] S Sankaranarayanan, S R Kandimalla, S Hasan, and others. 2020. Agentin-the-loop: Conversational agent support in service of reflection for learning during collaborative programming. Artif. Intell. (2020). https://link.springer.com/chapter/10.1007/978-3-030-52240-7_50 [72] Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? (Aug. 2022). arXiv:2208.06213 [cs.HC] http://arxiv.org/abs/2208.06213 [73] Noah L Schroeder, Olusola O Adesope, and Rachel Barouch Gilbert. 2013. How Effective are Pedagogical Agents for Learning? A MetaAnalytic Review. Journal of Educational Computing Research 49, 1 (July 2013), 1–39. https://doi.org/10.2190/EC.49.1.a [74] Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. Interactions 4, 6 (Nov. 1997), 42–61. https://doi.org/ 10.1145/267505.267514 [75] Alberto Sillitti, Giancarlo Succi, and Jelena Vlasenko. 2012. Understanding the impact of Pair Programming on developers attention: A case study on a large industrial experimentation. In 2012 34th International Conference on Software Engineering (ICSE) (Zurich). IEEE, 1094–1101. https://doi.org/10.1109/ICSE.2012.6227110 [76] Raymund Sison. 2009. Investigating the Effect of Pair Programming and Software Size on Software Quality and Programmer Productivity. In 2009 16th Asia-Pacific Software Engineering Conference. 187–193. https://doi.org/10.1109/APSEC.2009.71 [77] Joanna Smith, Joe Tessler, Elliot Kramer, and Calvin Lin. 2012. Using peer review to teach software testing. In Proceedings of the ninth annual international conference on International computing education research (Auckland, New Zealand) (ICER ’12). Association for Computing Machinery, New York, NY, USA, 93–98. https://doi.org/10.1145/ 2361276.2361295 [78] Shashank Sonkar, Lucy Liu, Debshila Basu Mallick, and Richard G Baraniuk. 2023. CLASS Meet SPOCK: An Education Tutoring Chatbot based on Learning Science Principles. (May 2023). arXiv:2305.13272 [cs.CL] http://arxiv.org/abs/2305.13272 [79] W Sun and G Marakas. 2009. The True Cost of Pair Programming: Development of a Comprehensive Model and Test. Americas Conference on Information Systems (2009). https://www.semanticscholar.org/ paper/647fc48650e4f19962c8a6feb87f3bdedde9dd04 [80] Chenhao Tan. 2023. On AI Anthropomorphism - HumanCentered AI - Medium. https://medium.com/human-centered-ai/onai-anthropomorphism-abff4cecc5ae. https://medium.com/humancentered-ai/on-ai-anthropomorphism-abff4cecc5ae Accessed: 2023-4- 23. [81] Lynda Thomas, Mark Ratcliffe, and Ann Robertson. 2003. Code warriors and code-a-phobes: a study in attitude and pair programming. SIGCSE Bull. 35, 1 (Jan. 2003), 363–367. https://doi.org/10.1145/792548. 612007 [82] H Holden Thorp. 2023. ChatGPT is fun, but not an author. Science 379, 6630 (Jan. 2023), 313. https://doi.org/10.1126/science.adg7879 [83] Karthikeyan Umapathy and Albert D Ritzhaupt. 2017. A Meta-Analysis of Pair-Programming in Computer Programming Courses: Implications for Educational Practice. ACM Trans. Comput. Educ. 17, 4 (Aug. 2017), 1–13. https://doi.org/10.1145/2996201 [84] Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22, Article 332). Association for Computing Machinery, New York, NY, USA, 1–7. https: //doi.org/10.1145/3491101.3519665 [85] Zichao Wang, Jakob Valdez, Debshila Basu Mallick, and Richard G Baraniuk. 2022. Towards Human-Like Educational Question Generation with Large Language Models. In Artificial Intelligence in Education. Springer International Publishing, 153–166. https://doi.org/10.1007/ 978-3-031-11644-5_13 [86] Laurie Williams and Richard L Upchurch. 2001. In support of student pair-programming. In Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education (Charlotte North Carolina USA). ACM, New York, NY, USA. https://doi.org/10.1145/364447. 364614 [87] Laurie Williams, Eric Wiebe, Kai Yang, Miriam Ferzli, and Carol Miller. 2002. In support of pair programming in the introductory computer science course. Comput. Sci. Educ. 12, 3 (Sept. 2002), 197–212. https: //doi.org/10.1076/csed.12.3.197.8618 [88] Dakota Wong, Austin Kothig, and Patrick Lam. 2022. Exploring the Verifiability of Code Generated by GitHub Copilot. ACM on Programming Languages (2022). https://www.semanticscholar.org/paper/ b5051fedaf17836f6b2a042cc4af4155159795c5 [89] Burak Yetiştiren, Işik Özsoy, and Eray Tüzün. 2022. Assessing the Quality of GitHub Copilot’s Code Generation. In 18th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE ’22). https://doi.org/10.1145/3558489.3559072 [90] Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming (San Diego CA USA). ACM, New York, NY, USA. https://doi.org/10.1145/3520312.3534864 Authors: (1) Qianou Ma (Corresponding author), Carnegie Mellon University, Pittsburgh, USA (qianouma@cmu.edu); (2) Tongshuang Wu, Carnegie Mellon University, Pittsburgh, USA (sherryw@cs.cmu.edu); (3) Kenneth Koedinger, Carnegie Mellon University, Pittsburgh, USA (koedinger@cmu.edu). Authors: Authors: (1) Qianou Ma (Corresponding author), Carnegie Mellon University, Pittsburgh, USA (qianouma@cmu.edu); (2) Tongshuang Wu, Carnegie Mellon University, Pittsburgh, USA (sherryw@cs.cmu.edu); (3) Kenneth Koedinger, Carnegie Mellon University, Pittsburgh, USA (koedinger@cmu.edu). This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv available on arxiv