Modash

modash

In this article, we’ll take a look at the world’s first Mobile DevOps, Performance, Productivity, and Maturity Assessment, or MODAS for short, and what it means for teams wanting to gain more insight into optimizing their development, testing, and deployment processes. The age of mobile innovation is upon us. We live in an unprecedented time of both technological opportunity and expectation. We have the ability to create all kinds of amazing solutions, but, at the same time, we have an enormous amount of room for improvement. But in order to improve, we need to know how we compare relative to others. And as you know, we can’t improve what we don’t measure, which I’m so excited to announce that has unveiled , which is a survey designed to provide insight and actionable guidance for mobile product organizations of all sizes and industries. Bitrise the world’s first Mobile DevOps Performance, Productivity, and Maturity Assessment (MODAS) The gist of MODAS is to provide the mobile community with a comprehensive comparative analysis of modern mobile development, testing, and delivery practices. Assessing DevOps maturity As I touched on above, the purpose of is to assess a mobile team’s Mobile Devops performance, productivity, and maturity. To do this, the survey has divided the mobile app delivery process into five stages. MODAS Creation Testing Deployment Monitoring Collaboration Each stage features a mandatory performance metric question, followed by optional clarifying questions and statements. The optional questions and statements will enable the survey both to gather more detailed data about how others in your industry are performing (and why) and to provide you with detailed recommendations about how to improve in each stage in the mobile delivery lifecycle. What’s in it for you? While the only takes a few minutes to complete, the impact on you and your team will be great and long-lasting. Once we’ve collected enough responses following the initial launch of MODAS, we’ll be able to use the data to benchmark your results against those of organizations similar to yours. assessment To wrap up the assessment, you’ll have the opportunity to submit your email along with your responses. This will allow Bitrise to follow up with you later, once the survey has hit a threshold of responses, with a benchmark of your results. It’s important to point out that your email will not be used for any marketing purposes. It will strictly be used to communicate your benchmarked results. The benchmarks you receive will help shed some light on the areas your organization has relative Mobile DevOps maturity, as well as highlighting areas that could use improvement. Along with providing you information on how you stack up in terms of Mobile DevOps maturity, Bitrise will also be launching a suite of resources to provide more information on you can improve in the areas that have been targeted for improvement. how So, if you’d like to gain more insight into how your mobile development, testing, monitoring, delivery, and collaborations stack up, you can jump into the now! MODAS survey Happy building! Also Published Here

Introducing the World’s First Mobile DevOps Assessment: MODAS

Controlling Automated Vehicles on Large Lane-free Roundabouts (Extended Version): References

Countering Mainstream Bias via End-to-End Adaptive Local Learning: Conclusion and References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012;
(2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden;
(3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland;
(4) Chetan Arora, Monash University, Melbourne, Australia;
(5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy;
(6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria;
(7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy;
(8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy;
(9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy;
(10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland;
(11) Mika Saari, Tampere University, Tampere, Finland;
(12)  Zheying Zhang, Tampere University, Tampere, Finland;
(13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction
Background
Research Approach
Research Agenda
Outlook and Conclusions
References 5. Outlook and Conclusions The field of GenAI in SE is an important and fast-moving SE research area, currently presenting opportunities as well as open challenges. GenAI has the power to redefine the way software is designed, developed, and maintained. The innovative capabilities of AI-driven systems promise to streamline processes, boost efficiency, and open up new avenues for creative problem-solving. However, there are also common challenges in adopting GenAI and specific challenges for SE activities, i.e., requirements engineering, software implementation, quality assurance, and SE education. Our research agenda, outlined in this paper, stands as one of the pioneering initiatives to systematically reveal the state of research as of Oct 2023. Software Engineering is a large spectrum that includes technical activities but also user-centric, process-driven and managerial aspects. A comprehensive view should consider insights and methodologies from research about software project development, software project management, professional competencies, software engineering education, and software businesses. Therefore, collaboration with experts from these diverse disciplines is essential to harness the full potential of this research area. In light of the evolving nature of GenAI research, significant work remains to establish GenAI in SE as a mature research area. Substantial theories must be developed to provide a solid theoretical underpinning for the practical applications. Technological advancements should be validated in a realistic project context. The adoption of GenAI in SE can also be combined with other active research areas, e.g., sustainability, trustworthy systems, and education. We acknowledge that this research agenda might still miss some important SE aspects where GenAI can be applied. RQs are presented to illustrate significant concerns in each SE area and are not meant to be exhaustive. The research agenda is open to additions of new tracks, topics, and RQs by other researchers interested in the research area. Through this collective effort and commitment from a diverse community of researchers, we can strengthen the foundations of GenAI in SE and ensure its relevance in a rapidly changing technological landscape. Declaration of Generative AI and AI-assisted technologies in the writing process During the preparation of this work, we used ChatGPT version 3.5 and Grammarly services to edit and improve our writing, i.e. rephrasing sentences and fixing typos and language mistakes. After using this tool/service, we reviewed and edited the content as needed and take(s) full responsibility for the content of the publication. This paper is available on arxiv under CC 4.0 license. Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction Background Research Approach Research Agenda Outlook and Conclusions References Abstract and Introduction Abstract and Introduction Background Background Research Approach Research Approach Research Agenda Research Agenda Outlook and Conclusions Outlook and Conclusions References References 5. Outlook and Conclusions The field of GenAI in SE is an important and fast-moving SE research area, currently presenting opportunities as well as open challenges. GenAI has the power to redefine the way software is designed, developed, and maintained. The innovative capabilities of AI-driven systems promise to streamline processes, boost efficiency, and open up new avenues for creative problem-solving. However, there are also common challenges in adopting GenAI and specific challenges for SE activities, i.e., requirements engineering, software implementation, quality assurance, and SE education. Our research agenda, outlined in this paper, stands as one of the pioneering initiatives to systematically reveal the state of research as of Oct 2023. Software Engineering is a large spectrum that includes technical activities but also user-centric, process-driven and managerial aspects. A comprehensive view should consider insights and methodologies from research about software project development, software project management, professional competencies, software engineering education, and software businesses. Therefore, collaboration with experts from these diverse disciplines is essential to harness the full potential of this research area. In light of the evolving nature of GenAI research, significant work remains to establish GenAI in SE as a mature research area. Substantial theories must be developed to provide a solid theoretical underpinning for the practical applications. Technological advancements should be validated in a realistic project context. The adoption of GenAI in SE can also be combined with other active research areas, e.g., sustainability, trustworthy systems, and education. We acknowledge that this research agenda might still miss some important SE aspects where GenAI can be applied. RQs are presented to illustrate significant concerns in each SE area and are not meant to be exhaustive. The research agenda is open to additions of new tracks, topics, and RQs by other researchers interested in the research area. Through this collective effort and commitment from a diverse community of researchers, we can strengthen the foundations of GenAI in SE and ensure its relevance in a rapidly changing technological landscape. Declaration of Generative AI and AI-assisted technologies in the writing process During the preparation of this work, we used ChatGPT version 3.5 and Grammarly services to edit and improve our writing, i.e. rephrasing sentences and fixing typos and language mistakes. After using this tool/service, we reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

Generative Artificial Intelligence for Software Engineering: Outlook and Conclusions

Generative Artificial Intelligence for Software Engineering: Background

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012;
(2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden;
(3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland;
(4) Chetan Arora, Monash University, Melbourne, Australia;
(5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy;
(6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria;
(7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy;
(8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy;
(9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy;
(10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland;
(11) Mika Saari, Tampere University, Tampere, Finland;
(12)  Zheying Zhang, Tampere University, Tampere, Finland;
(13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction
Background
Research Approach
Research Agenda
Outlook and Conclusions
References 3. Research Approach As indicated earlier, this paper aims to present a research agenda listing open Research Questions regarding GenAI in SE. We conducted a literature review (Section 3.1) and a series of focused groups to achieve this outcome (Section 3.2 and Section 3.3). The overall research process is presented in Figure 2. 3.1. Literature review A comprehensive and systematic review might not be suitable for research on GenAI in software development at the time this research being conducted. Firstly, research work is rapidly conducted and published on the topic, hence rendering the findings of a comprehensive review probably outdated shortly after publication. Secondly, we found a lot of relevant work as gray literature from non-traditional sources such as preprints, technical reports, and online forums. These sources may not be as rigorously reviewed or validated as peer-reviewed academic papers, making is difficult to assess their quality and reliability. Thirdly, we would like to publish the agenda as soon as possible to provide a reference for future research. A systematic literature review would consume extensive effort and time, which might be obsolete by the time the review is complete. Our strategy is to conduct focused, periodic reviews to capture the most current and relevant information without the extensive resource commitment of a comprehensive review. This approach allows for agility in keeping up with the latest developments without claiming comprehensiveness and repeatability. Our search approach involves two channels: • Online portals: we used Google Scholar and Scopus to search with a formulated search string • Gray literature sources: we search for papers from Arxiv and PaperwithCode • Forward and backward snowballing: we scan citations forward and backward from the articles included in our review. We used the search terms in Google Scholar and achieved the result (latest searched date October 2023). Google Scholar has the advantages of comprehensiveness and free access. It also covers grey literature where we found a significant number of research on GenAI at the searching time. 3.2. Focus groups We conducted four structured working sections as focus groups to identify, refine, and prioritize Research Questions on GenAI for SE. Focus groups have been used as a means to gather data in SE research [35, 36, 37, 38] Focus group sessions produce mainly qualitative information about the objects of study. The benefits of focus groups are that they produce insightful information, and the method is fairly inexpensive and fast to perform. This differs from a brainstorming section where participants are selected and navigated by a moderator, following a structured protocol so that the discussions stay focused [36, 39]. Kontio et al. also suggest online focus groups with a lot of advantages, such as group thinking, no travel cost, anonymous contribution, and support for large groups [36]. The value of the method is sensitive to the experience and insight of participants. Hence, we presented the information of the participants in Table 1. Focus groups’ timeline is from April 2023 to September 2023 (as shown in Figure 2). All participants are SE researchers who have experience or interest in the topic, as shown in Table 1. For each focus group, we developed a plan, which included the agenda of the section and a set of exercises for the participants. Each focus group lasted between 2 to 3 hours. To capture data from focus groups, we use moderator’s notes, Miro boards 7, and recording in case of online sessions. The focus of each focus group is different: • Focus group 1 (Exploratory brainstorming): Brainstorming was aimed at exploring ideas for potential opportunities and challenges when adopting GenAI in software development activities. We discussed a question, “Which SE areas will benefit from GenAI tools, such as ChatGPT and Copilot?”. We used SWEBOK areas to initialize the conversations. At the end of the group, 11 categories were created. Each category is assigned a section leader who is a co-author responsible for synthesizing content for the section. • Focus group 2 (Exploratory brainstorming): We discussed “What would be an interesting research topic for GenAI and Software Development and management?”. We initiated the discussion about “What could be interesting Research Questions for an empirical study on GenAI in SE?” Some questions were generated during the working session. • Focus group 3 (Validating brainstorming): prior to the meeting, a list of possible RQs was made. The list included 121 RQs. From this meeting, we aimed to validate the questions, if they are correctly and consistently interpreted, and if they are reasonable and meaningful to all participants of the meeting. Participants had sufficient time to review thoroughly and critically the RQs list. Each person left a comment for every question about their meaningfulness and feasibility. After this step, the RQs were revised and restructured. • Focus group 4 (Validating brainstorming): The discussion was conducted in subgroups, each group focused on one particular SE category. Prior to this session, the question list was finalized. In total, there are 78 RQs in 11 categories. RQs that are consensusly not practically important or meaningful are excluded. We also discussed and ranked RQs according to their novelty and timeliness of each RQs. However, we did not achieve consensus and a complete list of rankings for all RQs. Therefore, we decided not to present the ranking in this work. 3.3. Threats to validity According to Kontio et al. [37, 36]threats of the focus groups method in SE studies include group dynamics, social acceptability, hidden agenda, secrecy, and limited comprehension. To minimize the group dynamics threat, i.e., uncontrollable flow of discussion, all sections were coordinated by the main author, strictly following the pre-determined agenda with time control. Social acceptability can influence discussion results, i.e. participants contribute biased opinions because of positive or negative perceptions from others. We introduce clear rules for contribution at the beginning of focus groups. For online sessions, Miro is used so participants can anonymously provide their opinions. As time is relatively limited for an online focus group, some complex issues or points might not be not necessarily understood by all participants (limited comprehension). This threat is mitigated by organizing focus groups following up with each other. Between the sessions, participants were required to do their ”homework” so discussing points could be studied or reflected further beyond the scope of the focus groups. Four focus groups occurring in four months give sufficient time for individual and group reflection on the topic. This paper is available on arxiv under CC 4.0 license. Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction Background Research Approach Research Agenda Outlook and Conclusions References Abstract and Introduction Abstract and Introduction Background Background Research Approach Research Approach Research Agenda Research Agenda Outlook and Conclusions Outlook and Conclusions References References 3. Research Approach As indicated earlier, this paper aims to present a research agenda listing open Research Questions regarding GenAI in SE. We conducted a literature review (Section 3.1) and a series of focused groups to achieve this outcome (Section 3.2 and Section 3.3). The overall research process is presented in Figure 2. 3.1. Literature review 3.1. Literature review A comprehensive and systematic review might not be suitable for research on GenAI in software development at the time this research being conducted. Firstly, research work is rapidly conducted and published on the topic, hence rendering the findings of a comprehensive review probably outdated shortly after publication. Secondly, we found a lot of relevant work as gray literature from non-traditional sources such as preprints, technical reports, and online forums. These sources may not be as rigorously reviewed or validated as peer-reviewed academic papers, making is difficult to assess their quality and reliability. Thirdly, we would like to publish the agenda as soon as possible to provide a reference for future research. A systematic literature review would consume extensive effort and time, which might be obsolete by the time the review is complete. Our strategy is to conduct focused, periodic reviews to capture the most current and relevant information without the extensive resource commitment of a comprehensive review. This approach allows for agility in keeping up with the latest developments without claiming comprehensiveness and repeatability. Our search approach involves two channels: • Online portals: we used Google Scholar and Scopus to search with a formulated search string • Gray literature sources: we search for papers from Arxiv and PaperwithCode • Forward and backward snowballing: we scan citations forward and backward from the articles included in our review. We used the search terms in Google Scholar and achieved the result (latest searched date October 2023). Google Scholar has the advantages of comprehensiveness and free access. It also covers grey literature where we found a significant number of research on GenAI at the searching time. 3.2. Focus groups 3.2. Focus groups We conducted four structured working sections as focus groups to identify, refine, and prioritize Research Questions on GenAI for SE. Focus groups have been used as a means to gather data in SE research [35, 36, 37, 38] Focus group sessions produce mainly qualitative information about the objects of study. The benefits of focus groups are that they produce insightful information, and the method is fairly inexpensive and fast to perform. This differs from a brainstorming section where participants are selected and navigated by a moderator, following a structured protocol so that the discussions stay focused [36, 39]. Kontio et al. also suggest online focus groups with a lot of advantages, such as group thinking, no travel cost, anonymous contribution, and support for large groups [36]. The value of the method is sensitive to the experience and insight of participants. Hence, we presented the information of the participants in Table 1. Focus groups’ timeline is from April 2023 to September 2023 (as shown in Figure 2). All participants are SE researchers who have experience or interest in the topic, as shown in Table 1. For each focus group, we developed a plan, which included the agenda of the section and a set of exercises for the participants. Each focus group lasted between 2 to 3 hours. To capture data from focus groups, we use moderator’s notes, Miro boards 7, and recording in case of online sessions. The focus of each focus group is different: • Focus group 1 (Exploratory brainstorming): Brainstorming was aimed at exploring ideas for potential opportunities and challenges when adopting GenAI in software development activities. We discussed a question, “Which SE areas will benefit from GenAI tools, such as ChatGPT and Copilot?”. We used SWEBOK areas to initialize the conversations. At the end of the group, 11 categories were created. Each category is assigned a section leader who is a co-author responsible for synthesizing content for the section. • Focus group 2 (Exploratory brainstorming): We discussed “What would be an interesting research topic for GenAI and Software Development and management?”. We initiated the discussion about “What could be interesting Research Questions for an empirical study on GenAI in SE?” Some questions were generated during the working session. • Focus group 3 (Validating brainstorming): prior to the meeting, a list of possible RQs was made. The list included 121 RQs. From this meeting, we aimed to validate the questions, if they are correctly and consistently interpreted, and if they are reasonable and meaningful to all participants of the meeting. Participants had sufficient time to review thoroughly and critically the RQs list. Each person left a comment for every question about their meaningfulness and feasibility. After this step, the RQs were revised and restructured. • Focus group 4 (Validating brainstorming): The discussion was conducted in subgroups, each group focused on one particular SE category. Prior to this session, the question list was finalized. In total, there are 78 RQs in 11 categories. RQs that are consensusly not practically important or meaningful are excluded. We also discussed and ranked RQs according to their novelty and timeliness of each RQs. However, we did not achieve consensus and a complete list of rankings for all RQs. Therefore, we decided not to present the ranking in this work. 3.3. Threats to validity 3.3. Threats to validity According to Kontio et al. [37, 36]threats of the focus groups method in SE studies include group dynamics, social acceptability, hidden agenda, secrecy, and limited comprehension. To minimize the group dynamics threat, i.e., uncontrollable flow of discussion, all sections were coordinated by the main author, strictly following the pre-determined agenda with time control. Social acceptability can influence discussion results, i.e. participants contribute biased opinions because of positive or negative perceptions from others. We introduce clear rules for contribution at the beginning of focus groups. For online sessions, Miro is used so participants can anonymously provide their opinions. As time is relatively limited for an online focus group, some complex issues or points might not be not necessarily understood by all participants (limited comprehension). This threat is mitigated by organizing focus groups following up with each other. Between the sessions, participants were required to do their ”homework” so discussing points could be studied or reflected further beyond the scope of the focus groups. Four focus groups occurring in four months give sufficient time for individual and group reflection on the topic.

Generative Artificial Intelligence for Software Engineering: Research Approach

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012;
(2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden;
(3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland;
(4) Chetan Arora, Monash University, Melbourne, Australia;
(5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy;
(6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria;
(7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy;
(8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy;
(9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy;
(10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland;
(11) Mika Saari, Tampere University, Tampere, Finland;
(12)  Zheying Zhang, Tampere University, Tampere, Finland;
(13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction
Background
Research Approach
Research Agenda
Outlook and Conclusions
References Abstract Context Generative Artificial Intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI’s ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Objective. Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Method. We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for Software Engineering. Results. We identified 78 open Research Questions (RQs) in 11 areas of Software Engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision-making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Conclusions. GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research. Keywords: Generative Artificial Intelligence, GenAI, ChatGPT, CoPilot, Software Engineering, Software Project, Software Development, Focus Group, Literature review, Structured review, Literature survey, Research Agenda, Research Roadmap 1. Introduction In recent years, Generative Artificial Intelligence (GenAI) has attracted significant attention and interest from Software Engineering (SE) research and practice. GenAI tools such as GitHub Copilot1 and ChatGPT2 have been rapidly explored in various professional contexts given their remarkable performance in producing human-like content. The emerging adoption of  these tools reintroduces a host of classic concerns about productivity and quality when adopting new technologies. Clearly, code generation and test case optimization are SE tasks that directly benefit from recent large language models (LLMs) [1, 2, 3, 4]. Beyond traditional research on applied. Machine Learning (ML), GenAI tools introduced usability and accessibility, leveraging AI-generated content to a broader range of professionals, requiring less technical competence to work and integrate them into existing work environment. GenAI tools are also showing their potential in non-coding tasks, such as assisting requirements engineering, software processes and project management. At the moment, GenAI is an active research areas with several challenges and open questions. To do well on specific tasks, LLMs require fine-tuning or training. Research has much focused so far on achieving reliable and scalable GenAI output for different SE tasks. GenAI models are inherently nondeterministic: the same prompt produces different answers on different inference executions [5]. Moreover, GenAI’s output can be very sensitive to the input parameters or settings [6, 7]. Hallucination is another common concern with AI-generated content, as they can be imaginary or fictional [8]. The promise of AI automation might be closer than ever in SE [9], when these open challenges are sufficiently addressed. Research agenda is a popular type of work for guiding and organizing research efforts in certain research areas [10, 11, 12, 13, 14, 15]. It often includes a review of existing work, and a presentation of directions, visions and priorities, enabling researchers to make meaningful contributions to their respective fields and to address pressing challenges. This research agenda is driven by past and current work on GenAI for SE. Based on focus groups and a literature review, we portray the current research on GenAI in SE and present the research agenda with open challenges for future research. While we do not emphasize the comprehensiveness of the literature review due to the fast-moving nature of the topic, the focus groups present practical expectations of the future roles of GenAI in software development. The research agenda is organized into 11 areas of concern, which are: (1) Requirements Engineering, (2) Software Design,(3) Sofware Implementation, (4) Quality Assurance, (5) Software Maintenance and Evolution, (6) Software Processes and Tools, (7) Engineering Management, (8) Professional Competencies, (9) Software Engineering Education, (10) Macro Aspects and (11) Fundamental concerns of GenAI. Two international events have provided a ground for this work. As a part of the first international workshop on AI-assisted Agile Software Development3 , we ran our first focus group to explore the benefits and challenges of AI-assisted Agile software development. At the Requirements Engineering (RE) conference, we organized the second international workshop on Requirements Engineering for Software Startups and Emerging Technologies (RESET) 4 with a special theme on GenAI and Requirements Engineering. Consequently, the identified open challenges reflect not just the existing gaps in literature but also draw from the practical, value-driven insights of the co-authors. The paper is structured as follows: we clarify the meaning of GenAI and what we know about GenAI in SE in Section 2, and we will present our research approach in Section 3. Section 4 presents our Research Agenda. Finally, Section 5 is Outlook and Conclusions. This paper is available on arxiv under CC 4.0 license. Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction Background Research Approach Research Agenda Outlook and Conclusions References Abstract and Introduction Abstract and Introduction Background Background Research Approach Research Approach Research Agenda Research Agenda Outlook and Conclusions Outlook and Conclusions References References Abstract Context Generative Artificial Intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI’s ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Context Objective. Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Objective. Method . We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for Software Engineering. Method Results . We identified 78 open Research Questions (RQs) in 11 areas of Software Engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision-making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Results Conclusions. GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research. Conclusions. Keywords: Generative Artificial Intelligence, GenAI, ChatGPT, CoPilot, Software Engineering, Software Project, Software Development, Focus Group, Literature review, Structured review, Literature survey, Research Agenda, Research Roadmap Keywords: 1. Introduction In recent years, Generative Artificial Intelligence (GenAI) has attracted significant attention and interest from Software Engineering (SE) research and practice. GenAI tools such as GitHub Copilot1 and ChatGPT2 have been rapidly explored in various professional contexts given their remarkable performance in producing human-like content. The emerging adoption of  these tools reintroduces a host of classic concerns about productivity and quality when adopting new technologies. Clearly, code generation and test case optimization are SE tasks that directly benefit from recent large language models (LLMs) [1, 2, 3, 4]. Beyond traditional research on applied. Machine Learning (ML), GenAI tools introduced usability and accessibility, leveraging AI-generated content to a broader range of professionals, requiring less technical competence to work and integrate them into existing work environment. GenAI tools are also showing their potential in non-coding tasks, such as assisting requirements engineering, software processes and project management. At the moment, GenAI is an active research areas with several challenges and open questions. To do well on specific tasks, LLMs require fine-tuning or training. Research has much focused so far on achieving reliable and scalable GenAI output for different SE tasks. GenAI models are inherently nondeterministic: the same prompt produces different answers on different inference executions [5]. Moreover, GenAI’s output can be very sensitive to the input parameters or settings [6, 7]. Hallucination is another common concern with AI-generated content, as they can be imaginary or fictional [8]. The promise of AI automation might be closer than ever in SE [9], when these open challenges are sufficiently addressed. Research agenda is a popular type of work for guiding and organizing research efforts in certain research areas [10, 11, 12, 13, 14, 15]. It often includes a review of existing work, and a presentation of directions, visions and priorities, enabling researchers to make meaningful contributions to their respective fields and to address pressing challenges. This research agenda is driven by past and current work on GenAI for SE. Based on focus groups and a literature review, we portray the current research on GenAI in SE and present the research agenda with open challenges for future research. While we do not emphasize the comprehensiveness of the literature review due to the fast-moving nature of the topic, the focus groups present practical expectations of the future roles of GenAI in software development. The research agenda is organized into 11 areas of concern, which are: (1) Requirements Engineering, (2) Software Design,(3) Sofware Implementation, (4) Quality Assurance, (5) Software Maintenance and Evolution, (6) Software Processes and Tools, (7) Engineering Management, (8) Professional Competencies, (9) Software Engineering Education, (10) Macro Aspects and (11) Fundamental concerns of GenAI. Two international events have provided a ground for this work. As a part of the first international workshop on AI-assisted Agile Software Development3 , we ran our first focus group to explore the benefits and challenges of AI-assisted Agile software development. At the Requirements Engineering (RE) conference, we organized the second international workshop on Requirements Engineering for Software Startups and Emerging Technologies (RESET) 4 with a special theme on GenAI and Requirements Engineering. Consequently, the identified open challenges reflect not just the existing gaps in literature but also draw from the practical, value-driven insights of the co-authors. The paper is structured as follows: we clarify the meaning of GenAI and what we know about GenAI in SE in Section 2, and we will present our research approach in Section 3. Section 4 presents our Research Agenda. Finally, Section 5 is Outlook and Conclusions.

Generative Artificial Intelligence for Software Engineering: Abstract and Introduction

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 8 CONCLUSION Automated testing is crucial for helping improve app quality. Despite the dozens of mobile app GUI testing techniques, how to automatically generate the diversified unusual text inputs for fully testing mobile apps remains a challenge. This paper proposes InputBlaster which leverages the LLM to produce the unusual inputs together with the mutation rules which serve as the reasoning chains. It formulates the unusual inputs generation problem as a task of producing a set of test generators, each of which can yield a batch of unusual text inputs under the same mutation rule. The evaluation is conducted for both effectiveness and usefulness, with 136% higher bug detection rate than the best baselines, and uncovering 37 new crashes. In the future, we plan to further analyze the root causes and repair strategy of these input-related bugs, and design automated bug repair methods. REFERENCES [1] 2022. Crash bug text. https://www.theguardian.com/technology/iphone-crashbug-text imessage-ios. [2] 2022. Crash bug text in ios. https://tech.hindustantimes.com/tech/news/becareful-a-new-text bomb-is-making-whatsapp-crash-and-will-hang-yourphone-71599532897852.html. [3] 2022. F-droid. https://f-droid.org/. [4] Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training LLMs for project-specific code-summarization. ASE (2022). [5] Nadia Alshahwan and Mark Harman. 2011. Automated web application testing using search based software engineering. In ASE. IEEE, 3–12. [6] Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated concolic testing of smartphone apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. 1–11. [7] Yauhen Leanidavich Arnatovich, Minh Ngoc Ngo, Tan Hee Beng Kuan, and Charlie Soh. 2016. Achieving high code coverage in android ui testing via automated widget exercising. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC). IEEE, 193–200. [8] Yauhen Leanidavich Arnatovich, Lipo Wang, Ngoc Minh Ngo, and Charlie Soh. 2018. Mobolic: An automated approach to exercising mobile application GUIs using symbiosis of online testing technique and customated input generation. Software: Practice and Experience 48, 5 (2018), 1107 1142. [9] Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications. 641–660. [10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [12] Tianqin Cai, Zhao Zhang, and Ping Yang. 2020. Fastbot: A Multi-Agent ModelBased Test Generation System Beijing Bytedance Network Technology Co., Ltd.. In Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test. 93–96. [13] Chen Chen, Baojiang Cui, Jinxin Ma, Runpu Wu, Jianchao Guo, and Wenqian Liu. 2018. A systematic review of fuzzing techniques. Computers & Security 75 (2018), 118–137. [14] Taolue Chen, Alejandro Flores-Lamas, Matthew Hague, Zhilei Han, Denghang Hu, Shuanglong Kan, Anthony W Lin, Philipp Rümmer, and Zhilin Wu. 2022. Solving string constraints with Regex dependent functions through transducers with priorities and variables. Proceedings of the ACM on Programming Languages 6, POPL (2022), 1–31. [15] Taolue Chen, Matthew Hague, Jinlong He, Denghang Hu, Anthony Widjaja Lin, Philipp Rümmer, and Zhilin Wu. 2020. A decision procedure for path feasibility of string manipulating programs with integer data type. In Automated Technology for Verification and Analysis: 18th International Symposium, ATVA 2020, Hanoi, Vietnam, October 19–23, 2020, Proceedings. Springer, 325–342. [16] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. 2020. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems 33 (2020), 22243–22255. [17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). [18] Joel D Day, Thorsten Ehlers, Mitja Kulczynski, Florin Manea, Dirk Nowotka, and Danny Bøgsted Poulsen. 2019. On solving word equations using SAT. In Reachability Problems: 13th International Conference, RP 2019, Brussels, Belgium, September 11–13, 2019, Proceedings 13. Springer, 93–106. [19] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan,  Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In UIST. [20] Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023. Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT. ISSTA (2023). [21] Yinlin Deng, Chenyuan Yang, Anjiang Wei, and Lingming Zhang. 2022. Fuzzing deep-learning libraries via automated relational API inference. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 44–56. [22] Android Developers. 2012. Ui/application exerciser monkey. [23] Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of android apps. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 481–492. [24] Robert Feldt and Simon Poulding. 2013. Finding test data with specific properties via metaheuristic search. In 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 350–359. [25] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems 35 (2022), 30583–30598. [26] Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 269–280. [27] Yuyu He, Lei Zhang, Zhemin Yang, Yinzhi Cao, Keke Lian, Shuai Li, Wei Yang, Zhibo Zhang, Min Yang, Yuan Zhang, et al. 2020. TextExerciser: feedback-driven text input exercising for android applications. In 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 1071–1087. [28] Luk Holk, Petr Jank, Anthony W Lin, and Rmmer. 2017. String constraints with concatenation and transducers solved efficiently. Proceedings of the ACM on Programming Languages 2, POPL (2017), 1–32. [29] Yang Hu, Umair Z Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. 2019. Re factoring based program repair applied to programming assignments. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 388–398. [30] Text input. 2022. Introduction about text input on Android Developer website. https://developer.android.google.cn/reference/android/widget/EditText?hl=en. [31] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of Code Language Models on Automated Program Repair. ICSE (2023). [32] Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. ICSE (2023). [33] Adam Kiezun, Vijay Ganesh, Shay Artzi, Philip J Guo, Pieter Hooimeijer, and Michael D Ernst. 2013. HAMPI: A solver for word equations over strings, regular expressions, and context-free grammars. ACM Transactions on Software Engineering and Methodology (TOSEM) 21, 4 (2013), 1–28. [34] Sebastian Krings, Joshua Schmidt, Patrick Skowronek, Jannik Dunkelau, and Dierk Ehmke. 2020. Towards constraint logic programming over strings for test data generation. In Declarative Programming and Knowledge Management: Conference on Declarative Programming, DECLARE 2019, Unifying INAP, WLP, and WFLP, Cottbus, Germany, September 9–12, 2019, Revised Selected Papers 22. Springer, 139–159. [35] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. 2020. Reinforcement learning with augmented data. Advances in neural information processing systems 33 (2020), 19884–19895. [36] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pretrained Large Language Models. In ICSE. [37] Guodong Li and Indradeep Ghosh. 2013. PASS: String solving with parameterized array and interval automaton. In Hardware and Software: Verification and Testing: 9th International Haifa Verification Conference, HVC 2013, Haifa, Israel, November 5-7, 2013, Proceedings 9. Springer, 15–31. [38] Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. DroidBot: A Lightweight UI-Guided Test Input Generator for Android (ICSE-C ’17). [39] Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 23– 26. [40] Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: a deep learning-based approach to automated black-box Android app testing. In ASE. IEEE, 1070–1073. [41] Hongliang Liang, Xiaoxiao Pei, Xiaodong Jia, Wuwei Shen, and Jian Zhang. 2018. Fuzzing: State of the art. IEEE Transactions on Reliability 67, 3 (2018), 1199–1218. [42] Tianyi Liang, Andrew Reynolds, Cesare Tinelli, Clark Barrett, and Morgan Deters. 2014. A DPLL (T) theory solver for a theory of strings and regular expressions. In Computer Aided Verification: 26th International Conference, CAV 2014, Held as Part of the Vienna Summer of Logic, VSL 2014, Vienna, Austria, July 18-22, 2014. Proceedings 26. Springer, 646–662. [43] Peng Liu, Xiangyu Zhang, Marco Pistoia, Yunhui Zheng, Manoel Marques, and Lingfei Zeng. 2017. Automatic text input generation for mobile testing. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 643–653. [44] Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing. arXiv preprint arXiv:2212.04732 (2022). [45] Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl Eyes: Spotting UI Display Issues via Visual Understanding. In ASE. IEEE. https://doi.org/10.1145/3324884.3416547 [46] Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Nighthawk: Fully Automated Localizing UI Display Issues via Visual Understanding. IEEE Transactions on Software Engineering (2022), 1–16. https: //doi.org/10.1109/TSE.2022.3150876 [47] Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding. 48–55. [48] Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 224–234. [49] Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 94–105. [50] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. Computer Science (2013). [51] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv preprint arXiv:2202.12837 (2022). [52] Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-Based Prompt Selection for Code Related Few-Shot Learning. In Proceedings of the 45th International Conference on Software Engineering (ICSE’23). [53] Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153–164. [54] Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. ICLR (2022). [55] Simon Poulding and Robert Feldt. 2017. Generating controllably invalid and atypical inputs for robustness testing. In 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 81–84. [56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020), 140:1–140:67. [57] Vaibhav Rastogi, Yan Chen, and William Enck. 2013. Appsplayground: automatic security analysis of smartphone applications. In Proceedings of the third ACM conference on Data and application security and privacy. 209–220. [58] J Schulman, B Zoph, C Kim, J Hilton, J Menick, J Weng, JFC Uribe, L Fedus, L Metz, M Pokorny, et al. 2022. ChatGPT: Optimizing language models for dialogue. [59] Carolyn B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Transactions on software engineering 25, 4 (1999), 557–572. [60] Mike Sharples. 2022. Automated Essay Writing: An AIED Opinion. International Journal of Artificial Intelligence in Education (2022), 1–8. [61] Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 245–256. [62] Nezih Sunman, Yiğit Soydan, and Hasan Sözer. 2022. Automated web application testing driven by pre-recorded test cases. Journal of Systems and Software (2022), 111441. [63] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2014. S3: A symbolic string solver for vulnerability detection in web applications. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. 1232–1243. [64] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2017. Model counting for recursively-defined strings. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part II 30. Springer, 399–418. [65] UIAutomator. 2021. Python wrapper of Android uiautomator test tool. https://github. com/xiaocong/uiautomator. [66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems (2017). [67] Jue Wang, Yanyan Jiang, Chang Xu, Chun Cao, Xiaoxing Ma, and Jian Lu. 2020. Combodroid: generating high-quality test inputs for android apps via use case combinations. In ICSE. 469–480. [68] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledgebased vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 3081–3089. [69] Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. 2022. An extensive study on pre-trained models for program understanding and generation. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 39–51. [70] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui  Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022).

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Conclusion and References

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 7 RELATED WORK There have been many automated GUI testing techniques for mobile apps [7, 9, 12, 22, 23, 26, 39, 48, 49, 57, 61, 67], yet they mainly focus on how to plan the exploration paths to fully cover the app activities and states. There are also studies [27, 43, 44] that aim at generating valid inputs to pass the GUI pages and are used to enrich the automated testing tools for higher coverage. None of them can conduct the testing of text input widgets. Testing Related with Text Inputs. For Web apps, SWAT [5] and AWET [62] generated the unusual inputs based on the pre-defined template. ACTEve [6] and S3 [63] first used symbolic execution to extract input constraints in the source code and then employ a solver to generate the inputs. They need to analyze the web code and can’t be directly applied to Android apps which have quite different rendering mechanisms. In addition, some constraints are dynamically generated (as shown in Section 2.1.2), and couldn’t be extracted from the source code. There are some string analysis methods for generating the strings that violate the constraints (e.g., string length) [14, 15, 18, 28, 33, 34, 37, 42, 64]. Although they are effective for string constraints, yet the inputs of mobile apps are more diversified, and they cannot work well in our task. With the breakthrough of LLMs, studies have proposed to explore how LLMs can be used to assist developers in a variety of tasks, such as code generation [54, 69], program repair [29, 31, 52], and code summarization [4, 69]. There is also a growing trend of applying LLM for software testing, e.g., fuzzing deep learning libraries [20], unit test generation [36], bug reproduction [32], valid input generation [44], etc, and achieves significant performance improvement. This work explores a different task, i.e., unusual text input generation for mobile apps, which provides new insights into how LLM can enhance the software testing practice. LLM for Software Engineering.

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Related Work

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 6 DISCUSSION AND THREATS TO VALIDITY 6.1 Generality Across Platforms The primary idea of InputBlaster is to generate unusual inputs for text widgets with the context information when running the apps. Although we only experiment with Android mobile apps, since other platforms have these similar types of information, InputBlaster can be used to conduct the testing of input widgets for other platforms. We conduct a small-scale experiment for another two popular platforms, and experiment on 10 iOS apps with 15 bugs and 10 Web apps with 18 bugs, with details on our website. Results show that InputBlaster’s bug detection rate is 80% for iOS apps and 78% for Web apps within 30 minutes testing time. This further demonstrates the generality and usefulness of InputBlaster, and we will conduct more thorough experiments in the future. 6.2 Threats of Validity The first threat concerns the representativeness of the experimental apps. We have selected popular and active apps which can partially reduce this threat. The second threat relates to the baseline selection. Since there are hardly any existing approaches for the unusual input generation of mobile apps, we employ 18 approaches from various aspects for a thorough comparison. There are inputs generation techniques for Web apps [5, 6, 62, 63], yet because they need to analyze the web code which is different from mobile apps considering the different rendering mechanism, and cannot be directly applied in our task, hence we don’t include them as the baselines. The third threat is that we only focus on the crash bugs, since they cause more serious effects and can be automatically observed, and existing studies also only explore this type of bug [38, 40, 53]. The fourth threat might lie in the process of manual categorization in Section 2.1.2. The process involves multiple practitioners and double-checking for the final decision. Also note that, the derived categorization is only for illustration, rather than serving as the ground truth for evaluation. The Fifth threat may exist in the uncertainty of LLM output results. LLM may not generate the corresponding output as expected, and we also design in-context learning and feedback mechanisms to ensure the output format and content of LLM. Last but not least, InputBlaster gradually builds the example dataset (Section 3.3.1) as the test goes on. This indicates the performance can be influenced by the testing order, e.g., when arranged in the first place, the crash could not be detected, yet when arranged after 10 apps are tested, the crash can be revealed, since the example dataset has accumulated more knowledge. In this paper, we use a random order of the experimental apps and would explore more in the future.

Stories, News & Mentions about MODASH

Modash’s Stories on HackerNoon