Table Of Links Table Of Links 1 INTRODUCTION 1 INTRODUCTION 1 INTRODUCTION 2 BACKGROUND AND RELATED WORK 2 BACKGROUND AND RELATED WORK 2 BACKGROUND AND RELATED WORK 3 RESEARCH DESIGN 3 RESEARCH DESIGN 3 RESEARCH DESIGN 4 MAPPING STUDY RESULTS 4 MAPPING STUDY RESULTS 4 MAPPING STUDY RESULTS 5 SURVEY RESULTS 5 SURVEY RESULTS 5 SURVEY RESULTS 6 COMPARING THE STATE-OF-THE-ART AND THE PRACTITIONERS’ PERCEPTIONS 6 COMPARING THE STATE-OF-THE-ART AND THE PRACTITIONERS’ PERCEPTIONS 6 COMPARING THE STATE-OF-THE-ART AND THE PRACTITIONERS’ PERCEPTIONS 7 DISCUSSION 7 DISCUSSION 7 DISCUSSION 8 CONCLUSIONS AND ACKNOWLEDGMENTS 8 CONCLUSIONS AND ACKNOWLEDGMENTS 8 CONCLUSIONS AND ACKNOWLEDGMENTS REFERENCES REFERENCES REFERENCES 8 CONCLUSIONS AND ACKNOWLEDGMENTS In this paper, we conducted a systematic mapping study and a survey to provide an overview of the different research themes on Modern Code Reviews (MCR) and analyze the practitioners’ opinions on the importance of those themes. Based on the juxtaposition of these two perspectives on MCR research, we outline an agenda for future research on MCR that is based on the identified research gaps and the perceived importance by practitioners. We have extracted the research contributions from 244 primary studies and summarized 15 years of MCR research in evidence briefings that can contribute to the knowledge transfer from academic research to practitioners. The five main themes of MCR research are: (1) support systems for code reviews (SS), (2) impact of code reviews on product quality and human aspects (IOF), (3) modern code review process properties (CRP), (4) impact of software development processes, patch characteristics, and tools on modern code reviews (ION), and (5) human and organizational factors (HOF). We conducted a survey to collect practitioners’ opinions about 46 statements representing the research in the identified themes. As a result, we learned that practitioners are most positive about the CRP and IOF theme, with special focus on the impact of code reviews on product quality. However, these themes represent a minority of the reviewed MCR research (66 primary studies). At the same time, the respondents are most negative about human factor- (HOF) and support systems-related (SS) research, which constitute together a majority of the reviewed research (108 primary studies). These results indicate that there is a misalignment between the state-of-the-art and the themes deemed important by most respondents of our survey. In addition, we found that the research that has been perceived positively by practitioners is generally also more frequently cited, i.e., has a larger research impact. Finally, as there has been an increased interest in reviewing MCR research in recent years, we analyzed other systematic literature reviews and mapping studies. Due to the different research questions of the respective studies, there is a varying overlap of the reviewed primary studies. Still, we find our observations on the potential gaps in MCR research corroborated. Analyzing the data extracted from the reviewed primary studies and guided by the answers from the survey, we propose nineteen new research questions we deem worth investigating in future MCR research. ACKNOWLEDGMENTS ACKNOWLEDGMENTS We would like to acknowledge that this work was supported by the Knowledge Foundation through the projects SERT – Software Engineering ReThought and OSIR Open-source inspired reuse (reference number 20190081) at Blekinge Institute of Technology, Sweden. We would also like to acknowledge all practitioners who contributed to our investigation. REFERENCES [1] Everton LG Alves, Myoungkyu Song, Tiago Massoni, Patricia DL Machado, and Miryung Kim. 2017. Refactoring inspection support for manual refactoring edits. IEEE Transactions on Software Engineering 44, 4 (2017), 365–383. [2] Aybuke Aurum, Håkan Petersson, and Claes Wohlin. 2002. State-of-the-art: software inspections after 25 years. Software Testing, Verification and Reliability 12, 3 (2002), 133–154. [3] Alberto Bacchelli and Christian Bird. 2013. Expectations, Outcomes, and Challenges of Modern Code Review. In Proceedings International Conference on Software Engineering (San Francisco, CA, USA) (ICSE). IEEE, 712–721. [4] Deepika Badampudi, Ricardo Britto, and Michael Unterkalmsteiner. 2019. Modern Code Reviews - Preliminary Results of a Systematic Mapping Study. In Proceedings of the Evaluation and Assessment on Software Engineering (EASE). ACM, Copenhagen, Denmark, 340–345. [5] Deepika Badampudi, Michael Unterkalmsteiner, and Ricardo Britto. 2021. Evidence briefings on modern code reviews. https://doi.org/10.5281/zenodo.5093742 [6] Deepika Badampudi, MICHAEL UNTERKALMSTEINER, and RICARDO BRITTO. 2022. Data used in modern code review mapping study and survey. https://doi.org/10.5281/zenodo.7464947 [7] Gabriele Bavota and Barbara Russo. 2015. Four eyes are better than two: On the impact of code reviews on software quality. In International Conference on Software Maintenance and Evolution (ICSME). IEEE, 81–90. [8] Marten Brouwer. 1999. Q is accounting for tastes. Journal of Advertising Research 39, 2 (1999), 35–35. [9] Steven R Brown. 1993. A primer on Q methodology. Operant subjectivity 16, 3/4 (1993), 91–138. [10] Bill Brykczynski. 1999. A survey of software inspection checklists. ACM SIGSOFT Software Engineering Notes 24, 1 (1999), 82. [11] Bruno Cartaxo, Gustavo Pinto, Baldoino Fonseca, Márcio Ribeiro, Pedro Pinheiro, Sergio Soares, and Maria Teresa Baldassarre. 2019. Software Engineering Research Community Viewpoints on Rapid Reviews. In Proceedings of the 13th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) (ESEM ’19). [12] Bruno Cartaxo, Gustavo Pinto, Elton Vieira, and Sérgio Soares. 2016. Evidence briefings: Towards a medium to transfer knowledge from systematic reviews to practitioners. In Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement. 1–10. [13] Jeffrey C Carver, Oscar Dieste, Nicholas A Kraft, David Lo, and Thomas Zimmermann. 2016. How practitioners perceive the relevance of esem research. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10. [14] H Alperen Çetin, Emre Doğan, and Eray Tüzün. 2021. A review of code reviewer recommendation studies: Challenges and future directions. Science of Computer Programming (2021), 102652. [15] Zhiyuan Chen, Young-Woo Kwon, and Myoungkyu Song. 2018. Clone refactoring inspection by summarizing clone refactorings and detecting inconsistent changes during software evolution. Journal of Software: Evolution and Process 30, 10 (2018), e1951. [16] Flavia Coelho, Tiago Massoni, and Everton LG Alves. 2019. Refactoring-aware code review: A systematic mapping study. In 2019 IEEE/ACM 3rd International Workshop on Refactoring (IWoR). IEEE, 63–66. [17] D. S. Cruzes and T. Dyba. 2011. Recommended Steps for Thematic Synthesis in Software Engineering. In 2011 International Symposium on Empirical Software Engineering and Measurement. 275–284. [18] Nicole Davila and Ingrid Nunes. 2021. A systematic literature review and taxonomy of modern code review. Journal of Systems and Software (2021), 110951. [19] Charles H Davis and Carolyn Michelle. 2011. Q methodology in audience research: Bridging the qualitative/quantitative ‘divide’. Participations: Journal of Audience and Reception Studies 8, 2 (2011), 559–593. [20] M. E. Fagan. 1976. Design and code inspections to reduce errors in program development. IBM Systems Journal 15, 3 (1976), 182–211. [21] Xavier Franch, Daniel Mendez, Andreas Vogelsang, Rogardt Heldal, Eric Knauss, Marc Oriol, Guilherme Travassos, Jeffrey Clark Carver, and Thomas Zimmermann. 2020. How do Practitioners Perceive the Relevance of Requirements Engineering Research? IEEE Transactions on Software Engineering (2020). [22] Theresia Devi Indriasari, Andrew Luxton-Reilly, and Paul Denny. 2020. A Review of Peer Code Review in Higher Education. ACM Transactions on Computing Education (TOCE) 20, 3 (2020), 1–25. [23] Martin Ivarsson and Tony Gorschek. 2011. A method for evaluating rigor and industrial relevance of technology evaluations. Empirical Software Engineering 16, 3 (2011), 365–395. [24] Barbara A. Kitchenham and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE-2007-01. Software Engineering Group, Keele University and Department of Computer Science, University of Durham, United Kingdom. [25] Sami Kollanus and Jussi Koskinen. 2009. Survey of software inspection research. The Open Software Engineering Journal 3, 1 (2009). [26] Oliver Laitenberger and Jean-Marc DeBaud. 2000. An encompassing life cycle centric survey of software inspection. Journal of systems and software 50, 1 (2000), 5–31. [27] David Lo, Nachiappan Nagappan, and Thomas Zimmermann. 2015. How practitioners perceive the relevance of software engineering research. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 415–425. [28] F Macdonald, J Miller, A Brooks, M Roper, and M Wood. 1995. A review of tool support for software inspection. In Proceedings Seventh International Workshop on Computer-Aided Software Engineering. IEEE, 340–349. [29] Joseph Maxwell. 1992. Understanding and validity in qualitative research. Harvard educational review 62, 3 (1992), 279–301. [30] Sumaira Nazir, Nargis Fatima, and Suriayati Chuprat. 2020. Modern code review benefits-primary findings of a systematic literature review. In Proceedings of the 3rd International Conference on Software Engineering and Information Management. ACM, 210–215. [31] Kai Petersen and Cigdem Gencel. 2013. Worldviews, Research Methods, and their Relationship to Validity in Empirical Software Engineering Research. In Proceedings of the 2013 Joint Conference of the 23nd International Workshop on Software Measurement (IWSM) and the 8th International Conference on Software Process and Product Measurement. 81–89. [32] Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64 (2015), 1 – 18. [33] Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering 14, 2 (2009), 131–164. [34] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern Code Review: A Case Study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (Gothenburg, Sweden) (ICSE-SEIP ’18). ACM, New York, NY, USA, 181–190. [35] Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5 (2017), 3909–3943. [36] Dong Wang, Yuki Ueda, Raula Gaikovina Kula, Takashi Ishio, and Kenichi Matsumoto. 2019. The Evolution of Code Review Research: A Systematic Mapping Study. arXiv:1911.08816 [cs.SE] [37] Dong Wang, Yuki Ueda, Raula Gaikovina Kula, Takashi Ishio, and Kenichi Matsumoto. 2021. Can we benchmark Code Review studies? A systematic mapping study of methodology, dataset, and metric. Journal of Systems and Software (2021), 111009. [38] Roel Wieringa, Neil Maiden, Nancy Mead, and Colette Rolland. 2005. Requirements Engineering Paper Classification and Evaluation Criteria: A Proposal and a Discussion. Requir. Eng. 11, 1 (Dec. 2005), 102–107. [39] Claes Wohlin. 2014. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In Proceedings 18th International Conference on Evaluation and Assessment in Software Engineering (EASE). ACM, London, UK, 1–10. [40] Claes Wohlin, Per Runeson, Paulo Anselmo da Mota Silveira Neto, Emelie Engström, Ivan do Carmo Machado, and Eduardo Santana De Almeida. 2013. On the reliability of mapping studies in software engineering. Journal of Systems and Software 86, 10 (2013), 2594–2610. [41] Aiora Zabala and Unai Pascual. 2016. Bootstrapping Q methodology to improve the understanding of human perspectives. PloS one 11, 2 (2016), e0148087. MAPPING STUDY REFERENCES MAPPING STUDY REFERENCES [42] Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017. SentiCR: a customized sentiment analysis tool for code review interactions. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 106–111. [43] Wisam Haitham Abbood Al-Zubaidi, Patanamon Thongtanunam, Hoa Khanh Dam, Chakkrit Tantithamthavorn, and Aditya Ghose. 2020. Workload-Aware Reviewer Recommendation Using a Multi-Objective Search-Based Approach. Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/3416508.3417115 [44] Adam Alami, Marisa Leavitt Cohn, and Andrzej Wąsowski. 2019. Why does code review work for open source software communities?. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1073–1083. [45] Eman Abdullah AlOmar, Hussein AlRubaye, Mohamed Wiem Mkaouer, Ali Ouni, and Marouane Kessentini. 2021. Refactoring practices in the context of modern code review: An industrial case study at Xerox. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 348–357. [46] Everton LG Alves, Myoungkyu Song, and Miryung Kim. 2014. RefDistiller: a refactoring aware code review tool for inspecting manual refactoring edits. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 751–754. [47] Hirohisa Aman. 2013. 0-1 Programming Model-Based Method for Planning Code Review Using Bug Fix History. In 2013 20th Asia-Pacific Software Engineering Conference (APSEC), Vol. 2. IEEE, 37–42. [48] F. Armstrong, F. Khomh, and B. Adams. 2017. Broadcast vs. Unicast Review Technology: Does It Matter?. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). 219–229. [49] Sumit Asthana, Rahul Kumar, Ranjita Bhagwan, Christian Bird, Chetan Bansal, Chandra Maddila, Sonu Mehta, and B. Ashok. 2019. WhoDo: Automating Reviewer Suggestions at Scale. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 937–945. https: //doi.org/10.1145/3338906.3340449 [50] Jai Asundi and Rajiv Jayant. 2007. Patch review processes in open source software development communities: A comparative case study. In 2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07). IEEE, 166c–166c. [51] Krishna Teja Ayinala, Kwok Sun Cheng, Kwangsung Oh, and Myoungkyu Song. 2020. Tool Support for Code Change Inspection with Deep Learning in Evolving Software. In 2020 IEEE International Conference on Electro Information Technology (EIT). IEEE, 013–017. [52] Krishna Teja Ayinala, Kwok Sun Cheng, Kwangsung Oh, Teukseob Song, and Myoungkyu Song. 2020. Code Inspection Support for Recurring Changes with Deep Learning in Evolving Software. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). 931–942. https://doi.org/10.1109/COMPSAC48688.2020.0-149 [53] Muhammad Ilyas Azeem, Qiang Peng, and Qing Wang. 2020. Pull Request Prioritization Algorithm based on Acceptance and Response Probability. In 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS). 231–242. https://doi.org/10.1109/QRS51102.2020.00041 [54] Alberto Bacchelli and Christian Bird. 2013. Expectations, Outcomes, and Challenges of Modern Code Review. In Proceedings International Conference on Software Engineering (San Francisco, CA, USA) (ICSE). IEEE, 712–721. [55] Vipin Balachandran. 2013. Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In Proceedings International Conference on Software Engineering. IEEE, 931–940. [56] Vipin Balachandran. 2020. Reducing accidental clones using instant clone search in automatic code review. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 781–783. https://doi.org/10.1109/ ICSME46990.2020.00089 [57] Faruk Balcı, Dilruba Sultan Haliloğlu, Onur Şahin, Cankat Tilki, Mehmet Ata Yurtsever, and Eray Tüzün. 2021. Augmenting Code Review Experience Through Visualization. In 2021 Working Conference on Software Visualization (VISSOFT). IEEE, 110–114. [58] Mike Barnett, Christian Bird, Joao Brunet, and Shuvendu K Lahiri. 2015. Helping developers help themselves: Automatic decomposition of code review changesets. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 134–144. [59] Tobias Baum, Fabian Kortum, Kurt Schneider, Arthur Brack, and Jens Schauder. 2017. Comparing pre-commit reviews and post-commit reviews using process simulation. Journal of Software: Evolution and Process 29, 11 (2017), e1865. [60] Tobias Baum, Olga Liskin, Kai Niklas, and Kurt Schneider. 2016. A faceted classification scheme for change-based industrial code review processes. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 74–85. [61] Tobias Baum, Olga Liskin, Kai Niklas, and Kurt Schneider. 2016. Factors influencing code review processes in industry. In Proceedings of the 2016 24th acm sigsoft international symposium on foundations of software engineering. 85–96. [62] Tobias Baum and Kurt Schneider. 2016. On the need for a new generation of code review tools. In International Conference on Product-Focused Software Process Improvement. Springer, 301–308. [63] Tobias Baum, Kurt Schneider, and Alberto Bacchelli. 2017. On the Optimal Order of Reading Source Code Changes for Review. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 329–340. [64] Tobias Baum, Kurt Schneider, and Alberto Bacchelli. 2019. Associating working memory capacity and code change ordering with code review performance. Empirical Software Engineering 24, 4 (2019), 1762–1798. [65] O. Baysal, O. Kononenko, R. Holmes, and M. W. Godfrey. 2012. The Secret Life of Patches: A Firefox Case Study. In 2012 19th Working Conference on Reverse Engineering. 447–455. [66] Olga Baysal, Oleksii Kononenko, Reid Holmes, and Michael W Godfrey. 2016. Investigating technical and non-technical factors influencing modern code review. Empirical Software Engineering 21, 3 (2016), 932–959. [67] Andrew Begel and Hana Vrzakova. 2018. Eye Movements in Code Review. In Proceedings of the Workshop on Eye Movements in Programming (Warsaw, Poland) (EMIP ’18). ACM, New York, NY, USA, Article 5, 5 pages. [68] Moritz Beller, Alberto Bacchelli, Andy Zaidman, and Elmar Juergens. 2014. Modern code reviews in open-source projects: Which problems do they fix?. In Proceedings of the 11th working conference on mining software repositories. 202–211 [69] Mario Bernhart and Thomas Grechenig. 2013. On the understanding of programs with continuous code reviews. In 21st International Conference on Program Comprehension (ICPC). IEEE, 192–198. [70] Mario Bernhart, Andreas Mauczka, and Thomas Grechenig. 2010. Adopting code reviews for agile software development. In 2010 Agile Conference. IEEE, 44–47. [71] M. Bernhart, S. Strobl, A. Mauczka, and T. Grechenig. 2012. Applying Continuous Code Reviews in Airport Operations Software. In 2012 12th International Conference on Quality Software. 214–219. [72] Christian Bird, Trevor Carnahan, and Michaela Greiler. 2015. Lessons learned from building and deploying a code review analytics platform. In Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE, 191–201. [73] Amiangshu Bosu and Jeffrey C. Carver. 2012. Peer Code Review in Open Source Communities using Reviewboard. In Proceedings of the ACM 4th Annual Workshop on Evaluation and Usability of Programming Languages and Tools (Tucson, Arizona, USA) (PLATEAU ’12). ACM, New York, NY, USA, 17–24. [74] Amiangshu Bosu and Jeffrey C Carver. 2013. Impact of peer code review on peer impression formation: A survey. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 133–142. [75] Amiangshu Bosu and Jeffrey C Carver. 2014. Impact of developer reputation on code review outcomes in OSS projects: an empirical investigation. In Proceedings 8th international symposium on empirical software engineering and measurement. ACM, 33. [76] Amiangshu Bosu, Jeffrey C Carver, Christian Bird, Jonathan Orbeck, and Christopher Chockley. 2016. Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft. IEEE Transactions on Software Engineering 43, 1 (2016), 56–75. [77] Amiangshu Bosu, Jeffrey C Carver, Munawar Hafiz, Patrick Hilley, and Derek Janni. 2014. Identifying the characteristics of vulnerable code changes: An empirical study. In Proceedings 22nd International Symposium on Foundations of Software Engineering. ACM, 257–268. [78] Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of useful code reviews: An empirical study at microsoft. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 146–156. [79] Rodrigo Brito and Marco Tulio Valente. 2021. RAID: Tool support for refactoring-aware code reviews. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 265–275. [80] Fabio Calefato, Filippo Lanubile, Federico Maiorano, and Nicole Novielli. 2018. Sentiment polarity detection for software development. Empirical Software Engineering 23, 3 (2018), 1352–1382. [81] Nathan Cassee, Bogdan Vasilescu, and Alexander Serebrenik. 2020. The silent helper: the impact of continuous integration on code reviews. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 423–434. [82] Maria Caulo, Bin Lin, Gabriele Bavota, Giuseppe Scanniello, and Michele Lanza. 2020. Knowledge transfer in modern code review. In Proceedings of the 28th International Conference on Program Comprehension. 230–240. [83] Amudha Chandrika K. R. and J. Amudha. 2018. A fuzzy inference system to recommend skills for source code review using eye movement data. Journal of Intelligent and Fuzzy Systems 34, 3 (2018), 1743–1754. [84] Ashish Chopra, Morgan Mo, Samuel Dodson, Ivan Beschastnikh, Sidney S Fels, and Dongwook Yoon. 2021. "@ alex, this fixes# 9": Analysis of Referencing Patterns in Pull Request Discussions. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–25. [85] Moataz Chouchen, Ali Ouni, Raula Gaikovina Kula, Dong Wang, Patanamon Thongtanunam, Mohamed Wiem Mkaouer, and Kenichi Matsumoto. 2021. Anti-patterns in modern code review: Symptoms and prevalence. In 2021 IEEE international conference on software analysis, evolution and reengineering (SANER). IEEE, 531–535. [86] Moataz Chouchen, Ali Ouni, Mohamed Wiem Mkaouer, Raula Gaikovina Kula, and Katsuro Inoue. 2021. WhoReview: A multi-objective search-based approach for code reviewers recommendation in modern code review. Applied Soft Computing 100 (2021), 106908. https://doi.org/10.1016/j.asoc.2020.106908 [87] Aleksandr Chueshev, Julia Lawall, Reda Bendraou, and Tewfik Ziadi. 2020. Expanding the Number of Reviewers in Open-Source Projects by Recommending Appropriate Developers. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 499–510. https://doi.org/10.1109/ICSME46990.2020.00054 [88] Flávia Coelho, Nikolaos Tsantalis, Tiago Massoni, and Everton LG Alves. 2021. An Empirical Study on RefactoringInducing Pull Requests. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–12. [89] Atacílio Cunha, Tayana Conte, and Bruno Gadelha. 2021. Code Review is just reviewing code? A qualitative study with practitioners in industry. In Brazilian Symposium on Software Engineering. 269–274. [90] Jacek Czerwonka, Michaela Greiler, and Jack Tilford. 2015. Code Reviews Do Not Find Bugs: How the Current Code Review Best Practice Slows Us Down. In Proceedings of the 37th International Conference on Software Engineering - Volume 2 (Florence, Italy) (ICSE ’15). IEEE Press, 27–28. [91] Anastasia Danilova, Alena Naiakshina, Anna Rasgauski, and Matthew Smith. 2021. Code Reviewing as Methodology for Online Security Studies with Developers-A Case Study with Freelancers on Password Storage. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021). 397–416. [92] Manoel Limeira de Lima Júnior, Daricélio Moreira Soares, Alexandre Plastino, and Leonardo Murta. 2015. Developers assignment for analyzing pull requests. In Proceedings 30th Annual ACM Symposium on Applied Computing. ACM, 1567–1572. [93] Marco di Biase, Magiel Bruntink, and Alberto Bacchelli. 2016. A security perspective on code review: The case of chromium. In 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 21–30. [94] Marco di Biase, Magiel Bruntink, Arie van Deursen, and Alberto Bacchelli. 2019. The effects of change decomposition on code review—a controlled experiment. PeerJ Computer Science 5 (2019), e193. [95] Eduardo Witter dos Santos and Ingrid Nunes. 2017. Investigating the Effectiveness of Peer Code Review in Distributed Software Development. In Proceedings of the 31st Brazilian Symposium on Software Engineering (Fortaleza, CE, Brazil) (SBES’17). ACM, New York, NY, USA, 84–93. [96] Tobias Dürschmid. 2017. Continuous Code Reviews: A Social Coding tool for Code Reviews inside the IDE. In Companion to the first International Conference on the Art, Science and Engineering of Programming. ACM, 41. [97] Felipe Ebert, Fernando Castor, Nicole Novielli, and Alexander Serebrenik. 2017. Confusion detection in code reviews. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 549–553. [98] Felipe Ebert, Fernando Castor, Nicole Novielli, and Alexander Serebrenik. 2018. Communicative intention in code review questions. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 519–523. [99] Felipe Ebert, Fernando Castor, Nicole Novielli, and Alexander Serebrenik. 2021. An exploratory study on confusion in code reviews. Empirical Software Engineering 26, 1 (2021), 1–48. [100] Vasiliki Efstathiou and Diomidis Spinellis. 2018. Code review comments: language matters. In Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results. ACM, 69–72. [101] Carolyn D Egelman, Emerson Murphy-Hill, Elizabeth Kammer, Margaret Morrow Hodges, Collin Green, Ciera Jaspan, and James Lin. 2020. Predicting developers’ negative feelings about code review. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 174–185. [102] Ikram El Asri, Noureddine Kerzazi, Gias Uddin, Foutse Khomh, and MA Janati Idrissi. 2019. An empirical study of sentiments in code reviews. Information and Software Technology 114 (2019), 37–54. [103] Muntazir Fadhel and Emil Sekerinski. 2021. Striffs: Architectural Component Diagrams for Code Reviews. In 2021 International Conference on Code Quality (ICCQ). IEEE, 69–78. [104] George Fairbanks. 2019. Better Code Reviews With Design by Contract. IEEE Software 36, 6 (2019), 53–56. https: //doi.org/10.1109/MS.2019.2934192 [105] Yuanrui Fan, Xin Xia, David Lo, and Shanping Li. 2018. Early prediction of merged code changes to prioritize reviewing tasks. Empirical Software Engineering (2018), 1–48. [106] Mikołaj Fejzer, Piotr Przymus, and Krzysztof Stencel. 2018. Profile based recommendation of code reviewers. Journal of Intelligent Information Systems 50, 3 (2018), 597–619. [107] Isabella Ferreira, Jinghui Cheng, and Bram Adams. 2021. The" Shut the f** k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–35. [108] Wojciech Frącz and Jacek Dajda. 2017. Experimental Validation of Source Code Reviews on Mobile Devices. In International Conference on Computational Science and Its Applications. Springer, 533–547. [109] Leonardo B Furtado, Bruno Cartaxo, Christoph Treude, and Gustavo Pinto. 2020. How successful are open source contributions from countries with different levels of human development? IEEE Software 38, 2 (2020), 58–63. [110] Lorenzo Gasparini, Enrico Fregnan, Larissa Braz, Tobias Baum, and Alberto Bacchelli. 2021. ChangeViz: Enhancing the GitHub Pull Request Interface with Method Call Information. In 2021 Working Conference on Software Visualization (VISSOFT). IEEE, 115–119. [111] Xi Ge, Saurabh Sarkar, and Emerson Murphy-Hill. 2014. Towards refactoring-aware code review. In Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engineering. ACM, 99–102. [112] Çağdaş Evren Gerede and Zeki Mazan. 2018. Will it pass? Predicting the outcome of a source code review. Turkish Journal of Electrical Engineering & Computer Sciences 26, 3 (2018), 1343–1353. [113] Daniel M German, Gregorio Robles, Germán Poo-Caamaño, Xin Yang, Hajimu Iida, and Katsuro Inoue. 2018. " Was My Contribution Fairly Reviewed?" A Framework to Study the Perception of Fairness in Modern Code Reviews. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 523–534. [114] Mehdi Golzadeh, Alexandre Decan, and Tom Mens. 2019. On the Effect of Discussions on Pull Request Decisions.. In 18th Belgium-Netherlands Software Evolution Workshop (BENEVOL) [115] Jesus M Gonzalez-Barahona, Daniel Izquierdo-Cortazar, Gregorio Robles, and Alvaro del Castillo. 2014. Analyzing gerrit code review parameters with bicho. Electronic Communications of the EASST (2014). [116] Jesús M. González-Barahona, Daniel Izquierdo-Cortázar, Gregorio Robles, and Mario Gallegos. 2014. Code Review Analytics: WebKit as Case Study. In Open Source Software: Mobile Open Source Technologies. Springer, 1–10. [117] Tanay Gottigundala, Siriwan Sereesathien, and Bruno da Silva. 2021. Qualitatively Analyzing PR Rejection Reasons from Conversations in Open-Source Projects. In 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, 109–112. [118] Bo Guo, Young-Woo Kwon, and Myoungkyu Song. 2019. Decomposing composite changes for code review and regression test selection in evolving software. Journal of Computer Science and Technology 34, 2 (2019), 416–436. [119] DongGyun Han, Chaiyong Ragkhitwetsagul, Jens Krinke, Matheus Paixao, and Giovanni Rosa. 2020. Does code review really remove coding convention violations?. In 2020 IEEE 20th International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 43–53. [120] Xiaofeng Han, Amjed Tahir, Peng Liang, Steve Counsell, and Yajing Luo. 2021. Understanding code smell detection via code review: A study of the openstack community. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 323–334. [121] Quinn Hanam, Ali Mesbah, and Reid Holmes. 2019. Aiding code change understanding with semantic change impact analysis. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 202–212. [122] Christoph Hannebauer, Michael Patalas, Sebastian Stünkel, and Volker Gruhn. 2016. Automatically recommending code reviewers based on their expertise: An empirical comparison. In Proceedings 31st International Conference on Automated Software Engineering. ACM, 99–110. [123] Masum Hasan, Anindya Iqbal, Mohammad Rafid Ul Islam, A. J. M. Imtiajur Rahman, and Amiangshu Bosu. 2021. Using a Balanced Scorecard to Identify Opportunities to Improve Code Review Effectiveness: An Industrial Experience Report. 26, 6 (2021). https://doi.org/10.1007/s10664-021-10038-w [124] Florian Hauser, Stefan Schreistter, Rebecca Reuter, Jurgen Horst Mottok, Hans Gruber, Kenneth Holmqvist, and Nick Schorr. 2020. Code Reviews in C++ Preliminary Results from an Eye Tracking Study. In ACM Symposium on Eye Tracking Research and Applications. 1–5. [125] V. J. Hellendoorn, P. T. Devanbu, and A. Bacchelli. 2015. Will They Like This? Evaluating Code Contributions with Language Models. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. 157–167. [126] Vincent J. Hellendoorn, Jason Tsay, Manisha Mukherjee, and Martin Hirzel. 2021. Towards Automating Code Review at Scale. Association for Computing Machinery, New York, NY, USA, 1479–1482. https://doi.org/10.1145/3468264.3473134 [127] Austin Z Henley, KIvanç Muçlu, Maria Christakis, Scott D Fleming, and Christian Bird. 2018. CFar: A Tool to Increase Communication, Productivity, and Review Quality in Collaborative Code Reviews. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 157. [128] Martin Hentschel, Reiner Hähnle, and Richard Bubel. 2016. Can formal methods improve the efficiency of code reviews?. In International Conference on Integrated Formal Methods. Springer, 3–19. [129] Toshiki Hirao, Akinori Ihara, and Ken-ichi Matsumoto. 2015. Pilot study of collective decision-making in the code review process. In Proceedings 25th Annual International Conference on Computer Science and Software Engineering. IBM, 248–251. [130] Toshiki Hirao, Akinori Ihara, Yuki Ueda, Passakorn Phannachitta, and Ken-ichi Matsumoto. 2016. The impact of a low level of agreement among reviewers in a code review process. In IFIP International Conference on Open Source Systems. Springer, 97–110. [131] Toshiki Hirao, Raula Gaikovina Kula, Akinori Ihara, and Kenichi Matsumoto. 2019. Understanding developer commenting in code reviews. IEICE Transactions on Information and Systems 102, 12 (2019), 2423–2432. [132] Toshiki Hirao, Shane McIntosh, Akinori Ihara, and Kenichi Matsumoto. 2019. The Review Linkage Graph for Code Review Analytics: A Recovery Approach and Empirical Study. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 578–589. https: //doi.org/10.1145/3338906.3338949 [133] Gerard J Holzmann. 2010. SCRUB: a tool for code reviews. Innovations in Systems and Software Engineering 6, 4 (2010), 311–318. [134] Syeda Sumbul Hossain, Yeasir Arafat, Md Hossain, Md Arman, Anik Islam, et al. 2020. Measuring the effectiveness of software code review comments. In International Conference on Advances in Computing and Data Sciences. Springer, 247–257. [135] Dongyang Hu, Yang Zhang, Junsheng Chang, Gang Yin, Yue Yu, and Tao Wang. 2019. Multi-reviewing pull-requests: An exploratory study on GitHub OSS projects. Information and Software Technology 115 (2019), 1–4. [136] Yuan Huang, Nan Jia, Xiangping Chen, Kai Hong, and Zibin Zheng. 2018. Salient-class location: Help developers understand code change in code review. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 770–774. [137] Yuan Huang, Nan Jia, Xiangping Chen, Kai Hong, and Zibin Zheng. 2020. Code Review Knowledge Perception: Fusing Multi-Features for Salient-Class Location. IEEE Transactions on Software Engineering (2020). [138] Yu Huang, Kevin Leach, Zohreh Sharafi, Nicholas McKay, Tyler Santander, and Westley Weimer. 2020. Biases and differences in code review using medical imaging and eye-tracking: genders, humans, and machines. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 456–468. [139] M. Ichinco. 2014. Towards crowdsourced large-scale feedback for novice programmers. In 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 189–190. [140] Daniel Izquierdo, Jesus Gonzalez-Barahona, Lars Kurth, and Gregorio Robles. 2018. Software Development Analytics for Xen: Why and How. IEEE Software (2018). [141] Daniel Izquierdo-Cortazar, Lars Kurth, Jesus M Gonzalez-Barahona, Santiago Dueñas, and Nelson Sekitoleko. 2016. Characterization of the Xen project code review process: an experience report. In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, 386–390. [142] Daniel Izquierdo-Cortazar, Nelson Sekitoleko, Jesus M Gonzalez-Barahona, and Lars Kurth. 2017. Using Metrics to Track Code Review Performance. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering. ACM, 214–223. [143] Jing Jiang, Jin Cao, and Li Zhang. 2017. An empirical study of link sharing in review comments. In Software engineering and methodology for emerging domains. Springer, 101–114. [144] Jing Jiang, Jia-Huan He, and Xue-Yuan Chen. 2015. Coredevrec: Automatic core member recommendation for contribution evaluation. Journal of Computer Science and Technology 30, 5 (2015), 998–1016. [145] Jing Jiang, Yun Yang, Jiahuan He, Xavier Blanc, and Li Zhang. 2017. Who should comment on this pull request? Analyzing attributes for more accurate commenter recommendation in pull-based development. Information and Software Technology 84 (2017), 48–62. [146] Marian Jureczko, Łukasz Kajda, and Paweł Górecki. 2020. Code review effectiveness: an empirical study on selected factors influence. IET Software 14, 7 (2020), 794–805. [147] Akshay Kalyan, Matthew Chiam, Jing Sun, and Sathiamoorthy Manoharan. 2016. A collaborative code review platform for github. In 2016 21st International Conference on Engineering of Complex Computer Systems (ICECCS). IEEE, 191–196. [148] Ritu Kapur, Balwinder Sodhi, Poojith U Rao, and Shipra Sharma. 2021. Using Paragraph Vectors to improve our existing code review assisting tool-CRUSO. In 14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference). 1–11. [149] David Kavaler, Premkumar Devanbu, and Vladimir Filkov. 2019. Whom are you going to call? determinants of@- mentions in github discussions. Empirical Software Engineering 24, 6 (2019), 3904–3932. [150] Noureddine Kerzazi and Ikram El Asri. 2016. Who Can Help to Review This Piece of Code?. In Collaboration in a Hyperconnected World, Hamideh Afsarmanesh, Luis M. Camarinha-Matos, and António Lucas Soares (Eds.). Springer, 289–301. [151] Shivam Khandelwal, Sai Krishna Sripada, and Y. Raghu Reddy. 2017. Impact of Gamification on Code Review Process: An Experimental Study. In Proceedings of the 10th Innovations in Software Engineering Conference (Jaipur, India) (ISEC ’17). ACM, New York, NY, USA, 122–126. [152] Jungil Kim and Eunjoo Lee. 2018. Understanding Review Expertise of Developers: A Reviewer Recommendation Approach Based on Latent Dirichlet Allocation. Symmetry 10, 4 (2018), 114. [153] N. Kitagawa, H. Hata, A. Ihara, K. Kogiso, and K. Matsumoto. 2016. Code Review Participation: Game Theoretical Modeling of Reviewers in Gerrit Datasets. In 2016 IEEE/ACM Cooperative and Human Aspects of Software Engineering (CHASE). 64–67. [154] O. Kononenko, O. Baysal, and M. W. Godfrey. 2016. Code Review Quality: How Developers See It. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). 1028–1038. [155] Oleksii Kononenko, Olga Baysal, Latifa Guerrouj, Yaxin Cao, and Michael W Godfrey. 2015. Investigating code review quality: Do people and participation matter?. In International Conference on Software Maintenance and Evolution (ICSME). IEEE, 111–120. [156] O. Kononenko, T. Rose, O. Baysal, M. Godfrey, D. Theisen, and B. de Water. 2018. Studying Pull Request Merges: A Case Study of Shopify’s Active Merchant. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). 124–133. [157] V. Kovalenko and A. Bacchelli. 2018. Code Review for Newcomers: Is It Different?. In 2018 IEEE/ACM 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). 29–32. [158] Vladimir Kovalenko, Nava Tintarev, Evgeny Pasynkov, Christian Bird, and Alberto Bacchelli. 2018. Does reviewer recommendation help developers? IEEE Transactions on Software Engineering (2018). [159] Andrey Krutauz, Tapajit Dey, Peter C Rigby, and Audris Mockus. 2020. Do code review measures explain the incidence of post-release defects? Empirical Software Engineering 25, 5 (2020), 3323–3356. [160] Harsh Lal and Gaurav Pahwa. 2017. Code review analysis of software system using machine learning techniques. In Intelligent Systems and Control (ISCO), 2017 11th International Conference on. IEEE, 8–13. [161] Samuel Lehtonen and Timo Poranen. 2015. Metrics for Gerrit code review. In Proceedings of the 14th Symposium on Programming Languages and Software Tools (SPLST’15) (CEUR Workshop Proceedings, Vol. 1525). CEUR-WS.org, 31–45. [162] Heng-Yi Li, Shu-Ting Shi, Ferdian Thung, Xuan Huo, Bowen Xu, Ming Li, and David Lo. 2019. DeepReview: Automatic Code Review Using Deep Multi-instance Learning. In Advances in Knowledge Discovery and Data Mining, Qiang Yang, Zhi-Hua Zhou, Zhiguo Gong, Min-Ling Zhang, and Sheng-Jun Huang (Eds.). Springer International Publishing, Cham, 318–330. [163] Zhixing Li, Yue Yu, Gang Yin, Tao Wang, Qiang Fan, and Huaimin Wang. 2017. Automatic Classification of Review Comments in Pull-based Development Model.. In SEKE. 572–577. [164] Zhi-Xing Li, Yue Yu, Gang Yin, Tao Wang, and Huai-Min Wang. 2017. What are they talking about? analyzing code reviews in pull-based development model. Journal of Computer Science and Technology 32, 6 (2017), 1060–1075. [165] J. Liang and O. Mizuno. 2011. Analyzing Involvements of Reviewers through Mining a Code Review Repository. In 2011 Joint Conference of the 21st International Workshop on Software Measurement and the 6th International Conference on Software Process and Product Measurement. 126–132. [166] Zhifang Liao, Yanbing Li, Dayu He, Jinsong Wu, Yan Zhang, and Xiaoping Fan. 2017. Topic-based Integrator Matching for Pull Request. In Global Communications Conference. IEEE, 1–6. [167] Jakub Lipcak and Bruno Rossi. 2018. A large-scale study on source code reviewer recommendation. In 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 378–387. [168] Mingwei Liu, Xin Peng, Andrian Marcus, Christoph Treude, Xuefang Bai, Gang Lyu, Jiazhan Xie, and Xiaoxin Zhang. 2021. Learning-based extraction of first-order logic representations of API directives. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 491–502. [169] L. MacLeod, M. Greiler, M. Storey, C. Bird, and J. Czerwonka. 2018. Code Reviewing in the Trenches: Challenges and Best Practices. IEEE Software 35, 4 (2018), 34–42. [170] Michał Madera and Rafał Tomoń. 2017. A case study on machine learning model for code review expert system in software engineering. In 2017 Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE, 1357–1363. [171] Mika V Mäntylä and Casper Lassenius. 2008. What types of defects are really discovered in code reviews? IEEE Transactions on Software Engineering 35, 3 (2008), 430–448. [172] Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E Hassan. 2016. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering 21, 5 (2016), 2146–2189. [173] Massimiliano Menarini, Yan Yan, and William G Griswold. 2017. Semantics-assisted code review: An efficient tool chain and a user study. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 554–565. [174] Andrew Meneely, Alberto C Rodriguez Tejeda, Brian Spates, Shannon Trudeau, Danielle Neuberger, Katherine Whitlock, Christopher Ketant, and Kayla Davis. 2014. An empirical investigation of socio-technical code review metrics and security vulnerabilities. In Proceedings 6th International Workshop on Social Software Engineering. ACM, 37–44. [175] Benjamin S. Meyers, Nuthan Munaiah, Emily Prud’hommeaux, Andrew Meneely, Josephine Wolff, Cecilia Ovesdotter Alm, and Pradeep Murukannaiah. 2018. A dataset for identifying actionable feedback in collaborative software development. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 126–131. https://doi.org/10.18653/v1/P18-2021 [176] Ehsan Mirsaeedi and Peter C. Rigby. 2020. Mitigating Turnover with Code Review Recommendation: Balancing Expertise, Workload, and Knowledge Distribution. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’20). Association for Computing Machinery, New York, NY, USA, 1183–1195. https://doi.org/10.1145/3377811.3380335 [177] Rahul Mishra and Ashish Sureka. 2014. Mining peer code review system for computing effort and contribution metrics for patch reviewers. In 2014 IEEE 4th Workshop on Mining Unstructured Data (MUD). IEEE, 11–15. [178] Rodrigo Morales, Shane McIntosh, and Foutse Khomh. 2015. Do code review practices impact design quality? a case study of the qt, vtk, and itk projects. In 22nd International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 171–180. [179] Sebastian Müller, Michael Würsch, Thomas Fritz, and Harald C Gall. 2012. An approach for collaborative code reviews using multi-touch technology. In Proceedings of the 5th International Workshop on Co-operative and Human Aspects of Software Engineering. IEEE, 93–99. [180] Nuthan Munaiah, Benjamin S Meyers, Cecilia O Alm, Andrew Meneely, Pradeep K Murukannaiah, Emily Prud’hommeaux, Josephine Wolff, and Yang Yu. 2017. Natural language insights from code reviews that missed a vulnerability. In International Symposium on Engineering Secure Software and Systems. Springer, 70–86. [181] Yukasa Murakami, Masateru Tsunoda, and Hidetake Uwano. 2017. WAP: Does Reviewer Age Affect Code Review Performance?. In International Symposium on Software Reliability Engineering (ISSRE). IEEE, 164–169. [182] Emerson Murphy-Hill, Jillian Dicker, Margaret Morrow Hodges, Carolyn D Egelman, Ciera Jaspan, Lan Cheng, Elizabeth Kammer, Ben Holtz, Matt Jorde, Andrea Knight, and Collin Green. 2021. Engineering Impacts of Anonymous Author Code Review: A Field Experiment. IEEE Transactions on Software Engineering (2021), 1–1. https://doi.org/10. 1109/TSE.2021.3061527 [183] Reza Nadri, Gema Rodriguez-Perez, and Meiyappan Nagappan. 2021. Insights Into Nonmerged Pull Requests in GitHub: Is There Evidence of Bias Based on Perceptible Race? IEEE Softw. 38, 2 (2021), 51–57. [184] Aziz Nanthaamornphong and Apatta Chaisutanon. 2016. Empirical evaluation of code smells in open source projects: preliminary results. In Proceedings 1st International Workshop on Software Refactoring. ACM, 5–8. [185] Takuto Norikane, Akinori Ihara, and Kenichi Matsumoto. 2018. Do Review Feedbacks Influence to a Contributor’s Time Spent on OSS Projects?. In International Conference on Big Data, Cloud Computing, Data Science & Engineering (BCD). IEEE, 109–113. [186] Sebastiaan Oosterwaal, Arie van Deursen, Roberta Coelho, Anand Ashok Sawant, and Alberto Bacchelli. 2016. Visualizing code and coverage changes for code review. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 1038–1041. [187] Ali Ouni, Raula Gaikovina Kula, and Katsuro Inoue. 2016. Search-based peer reviewers recommendation in modern code review. In Proceedings International Conference on Software Maintenance and Evolution (ICSME). IEEE, 367–377. [188] Matheus Paixao, Jens Krinke, Donggyun Han, and Mark Harman. 2018. CROP: Linking code reviews to source code changes. In Proceedings of the 15th International Conference on Mining Software Repositories. 46–49. [189] Matheus Paixao, Jens Krinke, DongGyun Han, Chaiyong Ragkhitwetsagul, and Mark Harman. 2019. The impact of code review on architectural changes. IEEE Transactions on Software Engineering 47, 5 (2019), 1041–1059. [190] Matheus Paixao and Paulo Henrique Maia. 2019. Rebasing in code review considered harmful: A large-scale empirical investigation. In 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 45–55. [191] Matheus Paixão, Anderson Uchôa, Ana Carla Bibiano, Daniel Oliveira, Alessandro Garcia, Jens Krinke, and Emilio Arvonio. 2020. Behind the intents: An in-depth empirical study on software refactoring in modern code review. In Proceedings of the 17th International Conference on Mining Software Repositories. 125–136. [192] Thai Pangsakulyanont, Patanamon Thongtanunam, Daniel Port, and Hajimu Iida. 2014. Assessing MCR discussion usefulness using semantic similarity. In 2014 6th International Workshop on Empirical Software Engineering in Practice. IEEE, 49–54. [193] Sebastiano Panichella, Venera Arnaoudova, Massimiliano Di Penta, and Giuliano Antoniol. 2015. Would static analysis tools help developers with code reviews?. In 22nd International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 161–170. [194] Sebastiano Panichella and Nik Zaugg. 2020. An empirical investigation of relevant changes and automation needs in modern code review. Empirical Software Engineering 25, 6 (2020), 4833–4872. [195] Luca Pascarella, Davide Spadini, Fabio Palomba, Magiel Bruntink, and Alberto Bacchelli. 2018. Information needs in contemporary code review. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 135. [196] Rajshakhar Paul, Amiangshu Bosu, and Kazi Zakia Sultana. 2019. Expressions of sentiments during code reviews: Male vs. female. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 26–37. [197] Rajshakhar Paul, Asif Kamal Turzo, and Amiangshu Bosu. 2021. Why security defects go unnoticed during code reviews? a case-control study of the chromium os project. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1373–1385. [198] Zhenhui Peng, Jeehoon Yoo, Meng Xia, Sunghun Kim, and Xiaojuan Ma. 2018. Exploring How Software Developers Work with Mention Bot in GitHub. In Proceedings 6th International Symposium of Chinese CHI. ACM, 152–155. [199] Gustavo Pinto, Luiz Felipe Dias, and Igor Steinmacher. 2018. Who gets a patch accepted first? comparing the contributions of employees and volunteers. In 2018 IEEE/ACM 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, 110–113. [200] Felix Raab. 2011. Collaborative code reviews on interactive surfaces. In Proceedings of the 29th Annual European Conference on Cognitive Ergonomics. ACM, 263–264. [201] Janani Raghunathan, Lifei Liu, and Huzefa Hatimbhai Kagdi. 2018. Feedback topics in modern code review: Automatic identification and impact on changes. (2018) [202] Giuliano Ragusa and Henrique Henriques. 2018. Code review tool for Visual Programming Languages. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 287–288. [203] M. M. Rahman and C. K. Roy. 2017. Impact of Continuous Integration on Code Reviews. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 499–502. https://doi.org/10.1109/MSR.2017.39 [204] Mohammad Masudur Rahman, Chanchal K Roy, and Jason A Collins. 2016. CoRReCT: code reviewer recommendation in GitHub based on cross-project and technology experience. In Proceedings International Conference on Software Engineering Companion (ICSE-C). IEEE, 222–231. [205] Mohammad Masudur Rahman, Chanchal K Roy, and Raula G Kula. 2017. Predicting usefulness of code review comments using textual features and developer experience. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 215–226. [206] Achyudh Ram, Anand Ashok Sawant, Marco Castelluccio, and Alberto Bacchelli. 2018. What makes a code change easier to review: an empirical investigation on code change reviewability. In Proceedings Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 201–212. [207] Soumaya Rebai, Abderrahmen Amich, Somayeh Molaei, Marouane Kessentini, and Rick Kazman. 2020. Multi-Objective Code Reviewer Recommendations: Balancing Expertise, Availability and Collaborations. Automated Software Engg. 27, 3–4 (dec 2020), 301–328. https://doi.org/10.1007/s10515-020-00275-6 [208] Peter Rigby, Brendan Cleary, Frederic Painchaud, Margaret-Anne Storey, and Daniel German. 2012. Contemporary Peer Review in Action: Lessons from Open Source Development. IEEE Softw. 29, 6 (Nov. 2012), 56–61. [209] Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). ACM, New York, NY, USA, 202–212. [210] Shade Ruangwan, Patanamon Thongtanunam, Akinori Ihara, and Kenichi Matsumoto. 2018. The impact of human factors on the participation decision of reviewers in modern code review. Empirical Software Engineering (2018), 1–44. [211] Nafiz Sadman, Md Manjurul Ahsan, and M A Parvez Mahmud. 2020. ADCR: An Adaptive TOOL to select ”Appropriate Developer for Code Review” based on Code Context. In 2020 11th IEEE Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON). 0583–0591. https://doi.org/10.1109/UEMCON51285.2020.9298102 [212] Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern Code Review: A Case Study at Google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (Gothenburg, Sweden) (ICSE-SEIP ’18). ACM, New York, NY, USA, 181–190. [213] Nishrith Saini and Ricardo Britto. 2021. Using Machine Intelligence to Prioritise Code Review Requests. IEEE Press, 11–20. https://doi.org/10.1109/ICSE-SEIP52600.2021.00010 [214] Ronie Salgado and Alexandre Bergel. 2017. Pharo Git Thermite: A Visual Tool for Deciding to Weld a Pull Request. In Proceedings of the 12th edition of the International Workshop on Smalltalk Technologies. ACM, 1–6. [215] Mateus Santos, Josemar Caetano, Johnatan Oliveira, and Humberto T. Marques-Neto. 2018. Analyzing The Impact Of Feedback In GitHub On The Software Developer’s Mood. In International Conference on Software Engineering & Knowledge Engineering. ACM. [216] Bonita Sharif, Michael Falcone, and Jonathan I. Maletic. 2012. An Eye-Tracking Study on the Role of Scan Time in Finding Source Code Defects. In Proceedings of the Symposium on Eye Tracking Research and Applications (Santa Barbara, California) (ETRA ’12). ACM, New York, NY, USA, 381–384. [217] Shipra Sharma and Balwinder Sodhi. 2019. Using Stack Overflow content to assist in code review. Software: Practice and Experience 49, 8 (2019), 1255–1277. [218] Shu-Ting Shi, Ming Li, David Lo, Ferdian Thung, and Xuan Huo. 2019. Automatic Code Review by Learning the Revision of Source Code. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (Jul. 2019), 4910–4917. https://doi.org/10.1609/aaai.v33i01.33014910 [219] Junji Shimagaki, Yasutaka Kamei, Shane McIntosh, Ahmed E Hassan, and Naoyasu Ubayashi. 2016. A study of the quality-impacting practices of modern code review at sony mobile. In International Conference on Software Engineering Companion (ICSE-C). IEEE, 212–221. [220] Moran Shochat, Orna Raz, and Eitan Farchi. 2008. SeeCode–A Code Review Plug-in for Eclipse. In Haifa Verification Conference. Springer, 205–209. [221] Devarshi Singh, Varun Ramachandra Sekar, Kathryn T Stolee, and Brittany Johnson. 2017. Evaluating how static analysis tools can reduce code review effort. In Visual Languages and Human-Centric Computing (VL/HCC), 2017 IEEE Symposium on. IEEE, 101–105. [222] J. Siow, C. Gao, L. Fan, S. Chen, and Y. Liu. 2020. CORE: Automating Review Recommendation for Code Changes. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE Computer Society, Los Alamitos, CA, USA, 284–295. https://doi.org/10.1109/SANER48275.2020.9054794 [223] Daricélio M. Soares, Manoel L. de Lima Júnior, Alexandre Plastino, and Leonardo Murta. 2018. What factors influence the reviewer assignment to pull requests? Information and Software Technology 98 (2018), 32 – 43. [224] Davide Spadini, Maurício Aniche, Margaret-Anne Storey, Magiel Bruntink, and Alberto Bacchelli. 2018. When Testing Meets Code Review: Why and How Developers Review Tests. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). ACM, New York, NY, USA, 677–687. [225] Davide Spadini, Gül Çalikli, and Alberto Bacchelli. 2020. Primers or reminders? The effects of existing review comments on code review. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 1171–1182. [226] Davide Spadini, Fabio Palomba, Tobias Baum, Stefan Hanenberg, Magiel Bruntink, and Alberto Bacchelli. 2019. Test-driven code review: an empirical study. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1061–1072. [227] Kai Spohrer, Thomas Kude, Armin Heinzl, and Christoph Schmidt. 2013. Peer-based quality assurance in information systems development: A transactive memory perspective. (2013). [228] Panyawut Sri-iesaranusorn, Raula Gaikovina Kula, and Takashi Ishio. 2021. Does Code Review Promote Conformance? A Study of OpenStack Patches. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 444–448. [229] Miroslaw Staron, Mirosław Ochodek, Wilhelm Meding, and Ola Söder. 2020. Using machine learning to identify code fragments for manual review. In 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 513–516. [230] Anton Strand, Markus Gunnarson, Ricardo Britto, and Muhmmad Usman. 2020. Using a context-aware approach to recommend code reviewers: findings from an industrial case study. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice. ACM, 1–10. [231] Emre Sülün, Eray Tüzün, and Uğur Doğrusöz. 2019. Reviewer Recommendation Using Software Artifact Traceability Graphs. In Proceedings of the Fifteenth International Conference on Predictive Models and Data Analytics in Software Engineering (Recife, Brazil) (PROMISE’19). Association for Computing Machinery, New York, NY, USA, 66–75. https: //doi.org/10.1145/3345629.3345637 [232] Andrew Sutherland and Gina Venolia. 2009. Can peer code reviews be exploited for later information needs?. In 31st International Conference on Software Engineering-Companion. IEEE, 259–262. [233] Rajendran Swamidurai, Brad Dennis, and Uma Kannan. 2014. Investigating the impact of peer code review and pair programming on test-driven development. In IEEE SOUTHEASTCON 2014. IEEE, 1–5. [234] Yida Tao and Sunghun Kim. 2015. Partitioning composite code changes to facilitate code review. In Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE, 180–190. [235] K. Ayberk Tecimer, Eray Tüzün, Hamdi Dibeklioglu, and Hakan Erdogmus. 2021. Detection and Elimination of Systematic Labeling Bias in Code Reviewer Recommendation Systems. In Evaluation and Assessment in Software Engineering (Trondheim, Norway) (EASE 2021). Association for Computing Machinery, New York, NY, USA, 181–190. https://doi.org/10.1145/3463274.3463336 [236] Christopher Thompson and David Wagner. 2017. A large-scale study of modern code review and security in open source projects. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. 83–92. [237] Patanamon Thongtanunam and Ahmed E Hassan. 2020. Review dynamics and their impact on software quality. IEEE Transactions on Software Engineering 47, 12 (2020), 2698–2712. [238] Patanamon Thongtanunam, Raula Gaikovina Kula, Ana Erika Camargo Cruz, Norihiro Yoshida, and Hajimu Iida. 2014. Improving code review effectiveness through reviewer recommendations. In Proceedings 7th International Workshop on Cooperative and Human Aspects of Software Engineering. ACM, 119–122. [239] Patanamon Thongtanunam, Shane McIntosh, Ahmed E Hassan, and Hajimu Iida. 2015. Investigating code review practices in defective files: An empirical study of the qt system. In Proceedings 12th Working Conference on Mining Software Repositories. IEEE, 168–179. [240] Patanamon Thongtanunam, Shane McIntosh, Ahmed E. Hassan, and Hajimu Iida. 2016. Revisiting Code Ownership and Its Relationship with Software Quality in the Scope of Modern Code Review. In Proceedings of the 38th International Conference on Software Engineering (Austin, Texas) (ICSE ’16). ACM, New York, NY, USA, 1039–1050. [241] Patanamon Thongtanunam, Shane McIntosh, Ahmed E Hassan, and Hajimu Iida. 2018. Review participation in modern code review: An empirical study of the Android, Qt, and OpenStack projects (journal-first abstract). In Proceedings 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 475–475. [242] Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Raula Gaikovina Kula, Norihiro Yoshida, Hajimu Iida, and Ken-ichi Matsumoto. 2015. Who should review my code? A file location-based code-reviewer recommendation approach for modern code review. In Proceedings International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 141–150. [243] Patanamon Thongtanunam, Xin Yang, Norihiro Yoshida, Raula Gaikovina Kula, Ana Erika Camargo Cruz, Kenji Fujiwara, and Hajimu Iida. 2014. Reda: A web-based visualization tool for analyzing modern code review dataset. In Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on. IEEE, 605–608. [244] Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. 2021. Towards Automating Code Review Activities. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 163–174. https://doi.org/10.1109/ICSE43902.2021.00027 [245] Yuriy Tymchuk, Andrea Mocci, and Michele Lanza. 2015. Code review: Veni, vidi, vici. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 151–160. [246] Anderson Uchôa, Caio Barbosa, Daniel Coutinho, Willian Oizumi, Wesley KG Assunçao, Silvia Regina Vergilio, Juliana Alves Pereira, Anderson Oliveira, and Alessandro Garcia. 2021. Predicting design impactful changes in modern code review: A large-scale empirical study. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 471–482. [247] Anderson Uchôa, Caio Barbosa, Willian Oizumi, Publio Blenílio, Rafael Lima, Alessandro Garcia, and Carla Bezerra. 2020. How does modern code review impact software design degradation? an in-depth empirical study. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 511–522. [248] Yuki Ueda, Akinori Ihara, Takashi Ishio, Toshiki Hirao, and Kenichi Matsumoto. 2018. How are IF-Conditional Statements Fixed Through Peer CodeReview? IEICE TRANSACTIONS on Information and Systems 101, 11 (2018), 2720–2729. [249] Yuki Ueda, Akinori Ihara, Takashi Ishio, and Kennichi Matsumoto. 2018. Impact of coding style checker on code review-a case study on the openstack projects. In 2018 9th International Workshop on Empirical Software Engineering in Practice (IWESEP). IEEE, 31–36. [250] Yuki Ueda, Takashi Ishio, Akinori Ihara, and Kenichi Matsumoto. 2019. Mining source code improvement patterns from similar code review works. In 2019 IEEE 13th International Workshop on Software Clones (IWSC). IEEE, 13–19. [251] Naomi Unkelos-Shpigel and Irit Hadar. 2016. Lets Make it Fun: Gamifying and Formalizing Code Review. In Proceedings of the 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering. SCITEPRESSScience and Technology Publications, Lda, 391–395. [252] Hidetake Uwano, Masahide Nakamura, Akito Monden, and Ken-ichi Matsumoto. 2007. Exploiting Eye Movements for Evaluating Reviewer’s Performance in Software Review. IEICE Trans. Fundam. Electron. Commun. Comput. Sci. E90-A, 10 (Oct. 2007), 2290–2300. [253] Erik Van Der Veen, Georgios Gousios, and Andy Zaidman. 2015. Automatically prioritizing pull requests. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 357–361. [254] P. van Wesel, B. Lin, G. Robles, and A. Serebrenik. 2017. Reviewing Career Paths of the OpenStack Developers. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). 544–548. [255] Giovanni Viviani and Gail C Murphy. 2016. Removing stagnation from modern code review. In Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity. ACM, 43–44. [256] Hana Vrzakova, Andrew Begel, Lauri Mehtätalo, and Roman Bednarik. 2020. Affect recognition in code review: An in-situ biometric study of reviewer’s affect. Journal of Systems and Software 159 (2020), 110434. [257] Chen Wang, Xiaoyuan Xie, Peng Liang, and Jifeng Xuan. 2017. Multi-Perspective Visualization to Assist Code Change Review. In 2017 24th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 564–569. [258] Dong Wang, Raula Gaikovina Kula, Takashi Ishio, and Kenichi Matsumoto. 2021. Automatic patch linkage detection in code review using textual content and file location features. Information and Software Technology 139 (2021), 106637. [259] Dong Wang, Tao Xiao, Patanamon Thongtanunam, Raula Gaikovina Kula, and Kenichi Matsumoto. 2021. Understanding shared links and their intentions to meet information needs in modern code review. Empirical Software Engineering 26, 5 (2021), 1–32. [260] Min Wang, Zeqi Lin, Yanzhen Zou, and Bing Xie. 2019. Cora: Decomposing and describing tangled code changes for reviewer. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1050–1061. [261] Qingye Wang, Bowen Xu, Xin Xia, Ting Wang, and Shanping Li. 2019. Duplicate Pull Request Detection: When Time Matters. In Proceedings of the 11th Asia-Pacific Symposium on Internetware (Fukuoka, Japan) (Internetware ’19). Association for Computing Machinery, New York, NY, USA, Article 8, 10 pages. https://doi.org/10.1145/3361242. 3361254 [262] Song Wang, Chetan Bansal, Nachiappan Nagappan, and Adithya Abraham Philip. 2019. Leveraging change intents for characterizing and identifying large-review-effort changes. In Proceedings of the Fifteenth International Conference on Predictive Models and Data Analytics in Software Engineering. 46–55. [263] Yanqing Wang, Xiaolei Wang, Yu Jiang, Yaowen Liang, and Ying Liu. 2016. A code reviewer assignment model incorporating the competence differences and participant preferences. Foundations of Computing and Decision Sciences 41, 1 (2016), 77–91. [264] Ruiyin Wen, David Gilbert, Michael G Roche, and Shane McIntosh. 2018. BLIMP Tracer: Integrating Build Impact Analysis with Code Review. In Proceedings International Conference on Software Maintenance and Evolution (ICSME). IEEE, 685–694. [265] Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco A Gerosa. 2020. Effects of adopting code review bots on pull requests to OSS projects. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 1–11. [266] Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco A. Gerosa. 2020. What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers. In Proceedings of the 34th Brazilian Symposium on Software Engineering (Natal, Brazil) (SBES ’20). Association for Computing Machinery, New York, NY, USA, 457–462. https://doi.org/10.1145/3422392.3422459 [267] Mairieli Wessel, Igor Wiese, Igor Steinmacher, and Marco Aurelio Gerosa. 2021. Don’t Disturb Me: Challenges of Interacting with Software Bots on Open Source Software Projects. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–21. [268] Xin Xia, David Lo, Xinyu Wang, and Xiaohu Yang. 2015. Who should review this change?: Putting text and file location analyses together for more accurate recommendations. In Proceedings International Conference on Software Maintenance and Evolution (ICSME). IEEE, 261–270. [269] Zhenglin Xia, Hailong Sun, Jing Jiang, Xu Wang, and Xudong Liu. 2017. A hybrid approach to code reviewer recommendation with collaborative filtering. In 2017 6th International Workshop on Software Mining (SoftwareMining). IEEE, 24–31. [270] Cheng Yang, Xunhui Zhang, Lingbin Zeng, Qiang Fan, Gang Yin, and Huaimin Wang. 2017. An empirical study of reviewer recommendation in pull-based development model. In Proceedings 9th Asia-Pacific Symposium on Internetware. ACM, 14. [271] Cheng Yang, Xun-hui Zhang, Ling-bin Zeng, Qiang Fan, Tao Wang, Yue Yu, Gang Yin, and Huai-min Wang. 2018. RevRec: A two-layer reviewer recommendation algorithm in pull-based development model. Journal of Central South University 25, 5 (2018), 1129–1143. [272] Xin Yang. 2014. Social Network Analysis in Open Source Software Peer Review. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China) (FSE 2014). ACM, New York, NY, USA, 820–822. [273] Xin Ye. 2019. Learning to Rank Reviewers for Pull Requests. IEEE Access 7 (2019), 85382–85391. https://doi.org/10. 1109/ACCESS.2019.2925560 [274] Xin Ye, Yongjie Zheng, Wajdi Aljedaani, and Mohamed Wiem Mkaouer. 2021. Recommending Pull Request Reviewers Based on Code Changes. Soft Comput. 25, 7 (apr 2021), 5619–5632. https://doi.org/10.1007/s00500-020-05559-3 [275] Haochao Ying, Liang Chen, Tingting Liang, and Jian Wu. 2016. EARec: leveraging expertise and authority for pull-request reviewer recommendation in GitHub. In Proceedings 3rd International Workshop on CrowdSourcing in Software Engineering. ACM, 29–35. [276] Yue Yu, Huaimin Wang, Gang Yin, and Charles X Ling. 2014. Reviewer recommender of pull-requests in GitHub. In Proceedings International Conference on Software Maintenance and Evolution (ICSME). IEEE, 609–612. [277] Fiorella Zampetti, Gabriele Bavota, Gerardo Canfora, and Massimiliano Di Penta. 2019. A study on the interplay between pull request review and continuous integration builds. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 38–48. [278] Farida El Zanaty, Toshiki Hirao, Shane McIntosh, Akinori Ihara, and Kenichi Matsumoto. 2018. An empirical study of design discussions in code review. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ACM, 1–10. [279] Motahareh Bahrami Zanjani, Huzefa Kagdi, and Christian Bird. 2016. Automatically recommending peer reviewers in modern code review. IEEE Transactions on Software Engineering 42, 6 (2016), 530–543. [280] Tianyi Zhang, Myoungkyu Song, Joseph Pinedo, and Miryung Kim. 2015. Interactive code review for systematic changes. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 111–122. [281] Weifeng Zhang, Zhen Pan, and Ziyuan Wang. 2020. Prediction Method of Code Review Time Based on Hidden Markov Model. In Web Information Systems and Applications, Guojun Wang, Xuemin Lin, James Hendler, Wei Song, Zhuoming Xu, and Genggeng Liu (Eds.). Springer International Publishing, Cham, 168–175. [282] Xuesong Zhang, Bradley Dorn, William Jester, Jason Van Pelt, Guillermo Gaeta, and Daniel Firpo. 2011. Design and Implementation of Java Sniper: A Community-Based Software Code Review Web Solution. In 2011 44th Hawaii International Conference on System Sciences. IEEE, 1–10. [283] Y. Zhang, G. Yin, Y. Yu, and H. Wang. 2014. A Exploratory Study of @-Mention in GitHub’s Pull-Requests. In 2014 21st Asia-Pacific Software Engineering Conference, Vol. 1. 343–350. [284] Guoliang Zhao, Daniel Alencar da Costa, and Ying Zou. 2019. Improving the pull requests review process using learning-to-rank algorithms. Empirical Software Engineering (2019), 1–31. [285] Yanbing Li Yan Zhang Xiaoping Fan Jinsong Wu Zhifang Liao, ZeXuan Wu. 2020. Core-reviewer recommendation based on Pull Request topic model and collaborator social network. Soft Computing 24 (2020), pages5683–5693. Authors: DEEPIKA BADAMPUDI MICHAEL UNTERKALMSTEINER RICARDO BRITTO Authors: Authors: DEEPIKA BADAMPUDI MICHAEL UNTERKALMSTEINER RICARDO BRITTO DEEPIKA BADAMPUDI MICHAEL UNTERKALMSTEINER RICARDO BRITTO This paper is available on arxiv under CC BY-NC-SA 4.0 license. This paper is available on arxiv under CC BY-NC-SA 4.0 license. available on arxiv available on arxiv
