Hello! \n\nWe're a fully remote team, distributed team of ∼20 people on a mission to help every creator earn a living

Modash

modash

In this article, we’ll take a look at the world’s first Mobile DevOps, Performance, Productivity, and Maturity Assessment, or MODAS for short, and what it means for teams wanting to gain more insight into optimizing their development, testing, and deployment processes. The age of mobile innovation is upon us. We live in an unprecedented time of both technological opportunity and expectation. We have the ability to create all kinds of amazing solutions, but, at the same time, we have an enormous amount of room for improvement. But in order to improve, we need to know how we compare relative to others. And as you know, we can’t improve what we don’t measure, which I’m so excited to announce that has unveiled , which is a survey designed to provide insight and actionable guidance for mobile product organizations of all sizes and industries. Bitrise the world’s first Mobile DevOps Performance, Productivity, and Maturity Assessment (MODAS) The gist of MODAS is to provide the mobile community with a comprehensive comparative analysis of modern mobile development, testing, and delivery practices. Assessing DevOps maturity As I touched on above, the purpose of is to assess a mobile team’s Mobile Devops performance, productivity, and maturity. To do this, the survey has divided the mobile app delivery process into five stages. MODAS Creation Testing Deployment Monitoring Collaboration Each stage features a mandatory performance metric question, followed by optional clarifying questions and statements. The optional questions and statements will enable the survey both to gather more detailed data about how others in your industry are performing (and why) and to provide you with detailed recommendations about how to improve in each stage in the mobile delivery lifecycle. What’s in it for you? While the only takes a few minutes to complete, the impact on you and your team will be great and long-lasting. Once we’ve collected enough responses following the initial launch of MODAS, we’ll be able to use the data to benchmark your results against those of organizations similar to yours. assessment To wrap up the assessment, you’ll have the opportunity to submit your email along with your responses. This will allow Bitrise to follow up with you later, once the survey has hit a threshold of responses, with a benchmark of your results. It’s important to point out that your email will not be used for any marketing purposes. It will strictly be used to communicate your benchmarked results. The benchmarks you receive will help shed some light on the areas your organization has relative Mobile DevOps maturity, as well as highlighting areas that could use improvement. Along with providing you information on how you stack up in terms of Mobile DevOps maturity, Bitrise will also be launching a suite of resources to provide more information on you can improve in the areas that have been targeted for improvement. how So, if you’d like to gain more insight into how your mobile development, testing, monitoring, delivery, and collaborations stack up, you can jump into the now! MODAS survey Happy building! Also Published Here

Introducing the World’s First Mobile DevOps Assessment: MODAS

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012;
(2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden;
(3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland;
(4) Chetan Arora, Monash University, Melbourne, Australia;
(5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy;
(6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria;
(7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy;
(8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy;
(9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy;
(10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland;
(11) Mika Saari, Tampere University, Tampere, Finland;
(12)  Zheying Zhang, Tampere University, Tampere, Finland;
(13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction
Background
Research Approach
Research Agenda
Outlook and Conclusions
References 5. Outlook and Conclusions The field of GenAI in SE is an important and fast-moving SE research area, currently presenting opportunities as well as open challenges. GenAI has the power to redefine the way software is designed, developed, and maintained. The innovative capabilities of AI-driven systems promise to streamline processes, boost efficiency, and open up new avenues for creative problem-solving. However, there are also common challenges in adopting GenAI and specific challenges for SE activities, i.e., requirements engineering, software implementation, quality assurance, and SE education. Our research agenda, outlined in this paper, stands as one of the pioneering initiatives to systematically reveal the state of research as of Oct 2023. Software Engineering is a large spectrum that includes technical activities but also user-centric, process-driven and managerial aspects. A comprehensive view should consider insights and methodologies from research about software project development, software project management, professional competencies, software engineering education, and software businesses. Therefore, collaboration with experts from these diverse disciplines is essential to harness the full potential of this research area. In light of the evolving nature of GenAI research, significant work remains to establish GenAI in SE as a mature research area. Substantial theories must be developed to provide a solid theoretical underpinning for the practical applications. Technological advancements should be validated in a realistic project context. The adoption of GenAI in SE can also be combined with other active research areas, e.g., sustainability, trustworthy systems, and education. We acknowledge that this research agenda might still miss some important SE aspects where GenAI can be applied. RQs are presented to illustrate significant concerns in each SE area and are not meant to be exhaustive. The research agenda is open to additions of new tracks, topics, and RQs by other researchers interested in the research area. Through this collective effort and commitment from a diverse community of researchers, we can strengthen the foundations of GenAI in SE and ensure its relevance in a rapidly changing technological landscape. Declaration of Generative AI and AI-assisted technologies in the writing process During the preparation of this work, we used ChatGPT version 3.5 and Grammarly services to edit and improve our writing, i.e. rephrasing sentences and fixing typos and language mistakes. After using this tool/service, we reviewed and edited the content as needed and take(s) full responsibility for the content of the publication. This paper is available on arxiv under CC 4.0 license. Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction Background Research Approach Research Agenda Outlook and Conclusions References Abstract and Introduction Abstract and Introduction Background Background Research Approach Research Approach Research Agenda Research Agenda Outlook and Conclusions Outlook and Conclusions References References 5. Outlook and Conclusions The field of GenAI in SE is an important and fast-moving SE research area, currently presenting opportunities as well as open challenges. GenAI has the power to redefine the way software is designed, developed, and maintained. The innovative capabilities of AI-driven systems promise to streamline processes, boost efficiency, and open up new avenues for creative problem-solving. However, there are also common challenges in adopting GenAI and specific challenges for SE activities, i.e., requirements engineering, software implementation, quality assurance, and SE education. Our research agenda, outlined in this paper, stands as one of the pioneering initiatives to systematically reveal the state of research as of Oct 2023. Software Engineering is a large spectrum that includes technical activities but also user-centric, process-driven and managerial aspects. A comprehensive view should consider insights and methodologies from research about software project development, software project management, professional competencies, software engineering education, and software businesses. Therefore, collaboration with experts from these diverse disciplines is essential to harness the full potential of this research area. In light of the evolving nature of GenAI research, significant work remains to establish GenAI in SE as a mature research area. Substantial theories must be developed to provide a solid theoretical underpinning for the practical applications. Technological advancements should be validated in a realistic project context. The adoption of GenAI in SE can also be combined with other active research areas, e.g., sustainability, trustworthy systems, and education. We acknowledge that this research agenda might still miss some important SE aspects where GenAI can be applied. RQs are presented to illustrate significant concerns in each SE area and are not meant to be exhaustive. The research agenda is open to additions of new tracks, topics, and RQs by other researchers interested in the research area. Through this collective effort and commitment from a diverse community of researchers, we can strengthen the foundations of GenAI in SE and ensure its relevance in a rapidly changing technological landscape. Declaration of Generative AI and AI-assisted technologies in the writing process During the preparation of this work, we used ChatGPT version 3.5 and Grammarly services to edit and improve our writing, i.e. rephrasing sentences and fixing typos and language mistakes. After using this tool/service, we reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

Generative Artificial Intelligence for Software Engineering: Outlook and Conclusions

Generative Artificial Intelligence for Software Engineering: Background

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012;
(2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden;
(3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland;
(4) Chetan Arora, Monash University, Melbourne, Australia;
(5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy;
(6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria;
(7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy;
(8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy;
(9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy;
(10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland;
(11) Mika Saari, Tampere University, Tampere, Finland;
(12)  Zheying Zhang, Tampere University, Tampere, Finland;
(13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction
Background
Research Approach
Research Agenda
Outlook and Conclusions
References 3. Research Approach As indicated earlier, this paper aims to present a research agenda listing open Research Questions regarding GenAI in SE. We conducted a literature review (Section 3.1) and a series of focused groups to achieve this outcome (Section 3.2 and Section 3.3). The overall research process is presented in Figure 2. 3.1. Literature review A comprehensive and systematic review might not be suitable for research on GenAI in software development at the time this research being conducted. Firstly, research work is rapidly conducted and published on the topic, hence rendering the findings of a comprehensive review probably outdated shortly after publication. Secondly, we found a lot of relevant work as gray literature from non-traditional sources such as preprints, technical reports, and online forums. These sources may not be as rigorously reviewed or validated as peer-reviewed academic papers, making is difficult to assess their quality and reliability. Thirdly, we would like to publish the agenda as soon as possible to provide a reference for future research. A systematic literature review would consume extensive effort and time, which might be obsolete by the time the review is complete. Our strategy is to conduct focused, periodic reviews to capture the most current and relevant information without the extensive resource commitment of a comprehensive review. This approach allows for agility in keeping up with the latest developments without claiming comprehensiveness and repeatability. Our search approach involves two channels: • Online portals: we used Google Scholar and Scopus to search with a formulated search string • Gray literature sources: we search for papers from Arxiv and PaperwithCode • Forward and backward snowballing: we scan citations forward and backward from the articles included in our review. We used the search terms in Google Scholar and achieved the result (latest searched date October 2023). Google Scholar has the advantages of comprehensiveness and free access. It also covers grey literature where we found a significant number of research on GenAI at the searching time. 3.2. Focus groups We conducted four structured working sections as focus groups to identify, refine, and prioritize Research Questions on GenAI for SE. Focus groups have been used as a means to gather data in SE research [35, 36, 37, 38] Focus group sessions produce mainly qualitative information about the objects of study. The benefits of focus groups are that they produce insightful information, and the method is fairly inexpensive and fast to perform. This differs from a brainstorming section where participants are selected and navigated by a moderator, following a structured protocol so that the discussions stay focused [36, 39]. Kontio et al. also suggest online focus groups with a lot of advantages, such as group thinking, no travel cost, anonymous contribution, and support for large groups [36]. The value of the method is sensitive to the experience and insight of participants. Hence, we presented the information of the participants in Table 1. Focus groups’ timeline is from April 2023 to September 2023 (as shown in Figure 2). All participants are SE researchers who have experience or interest in the topic, as shown in Table 1. For each focus group, we developed a plan, which included the agenda of the section and a set of exercises for the participants. Each focus group lasted between 2 to 3 hours. To capture data from focus groups, we use moderator’s notes, Miro boards 7, and recording in case of online sessions. The focus of each focus group is different: • Focus group 1 (Exploratory brainstorming): Brainstorming was aimed at exploring ideas for potential opportunities and challenges when adopting GenAI in software development activities. We discussed a question, “Which SE areas will benefit from GenAI tools, such as ChatGPT and Copilot?”. We used SWEBOK areas to initialize the conversations. At the end of the group, 11 categories were created. Each category is assigned a section leader who is a co-author responsible for synthesizing content for the section. • Focus group 2 (Exploratory brainstorming): We discussed “What would be an interesting research topic for GenAI and Software Development and management?”. We initiated the discussion about “What could be interesting Research Questions for an empirical study on GenAI in SE?” Some questions were generated during the working session. • Focus group 3 (Validating brainstorming): prior to the meeting, a list of possible RQs was made. The list included 121 RQs. From this meeting, we aimed to validate the questions, if they are correctly and consistently interpreted, and if they are reasonable and meaningful to all participants of the meeting. Participants had sufficient time to review thoroughly and critically the RQs list. Each person left a comment for every question about their meaningfulness and feasibility. After this step, the RQs were revised and restructured. • Focus group 4 (Validating brainstorming): The discussion was conducted in subgroups, each group focused on one particular SE category. Prior to this session, the question list was finalized. In total, there are 78 RQs in 11 categories. RQs that are consensusly not practically important or meaningful are excluded. We also discussed and ranked RQs according to their novelty and timeliness of each RQs. However, we did not achieve consensus and a complete list of rankings for all RQs. Therefore, we decided not to present the ranking in this work. 3.3. Threats to validity According to Kontio et al. [37, 36]threats of the focus groups method in SE studies include group dynamics, social acceptability, hidden agenda, secrecy, and limited comprehension. To minimize the group dynamics threat, i.e., uncontrollable flow of discussion, all sections were coordinated by the main author, strictly following the pre-determined agenda with time control. Social acceptability can influence discussion results, i.e. participants contribute biased opinions because of positive or negative perceptions from others. We introduce clear rules for contribution at the beginning of focus groups. For online sessions, Miro is used so participants can anonymously provide their opinions. As time is relatively limited for an online focus group, some complex issues or points might not be not necessarily understood by all participants (limited comprehension). This threat is mitigated by organizing focus groups following up with each other. Between the sessions, participants were required to do their ”homework” so discussing points could be studied or reflected further beyond the scope of the focus groups. Four focus groups occurring in four months give sufficient time for individual and group reflection on the topic. This paper is available on arxiv under CC 4.0 license. Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction Background Research Approach Research Agenda Outlook and Conclusions References Abstract and Introduction Abstract and Introduction Background Background Research Approach Research Approach Research Agenda Research Agenda Outlook and Conclusions Outlook and Conclusions References References 3. Research Approach As indicated earlier, this paper aims to present a research agenda listing open Research Questions regarding GenAI in SE. We conducted a literature review (Section 3.1) and a series of focused groups to achieve this outcome (Section 3.2 and Section 3.3). The overall research process is presented in Figure 2. 3.1. Literature review 3.1. Literature review A comprehensive and systematic review might not be suitable for research on GenAI in software development at the time this research being conducted. Firstly, research work is rapidly conducted and published on the topic, hence rendering the findings of a comprehensive review probably outdated shortly after publication. Secondly, we found a lot of relevant work as gray literature from non-traditional sources such as preprints, technical reports, and online forums. These sources may not be as rigorously reviewed or validated as peer-reviewed academic papers, making is difficult to assess their quality and reliability. Thirdly, we would like to publish the agenda as soon as possible to provide a reference for future research. A systematic literature review would consume extensive effort and time, which might be obsolete by the time the review is complete. Our strategy is to conduct focused, periodic reviews to capture the most current and relevant information without the extensive resource commitment of a comprehensive review. This approach allows for agility in keeping up with the latest developments without claiming comprehensiveness and repeatability. Our search approach involves two channels: • Online portals: we used Google Scholar and Scopus to search with a formulated search string • Gray literature sources: we search for papers from Arxiv and PaperwithCode • Forward and backward snowballing: we scan citations forward and backward from the articles included in our review. We used the search terms in Google Scholar and achieved the result (latest searched date October 2023). Google Scholar has the advantages of comprehensiveness and free access. It also covers grey literature where we found a significant number of research on GenAI at the searching time. 3.2. Focus groups 3.2. Focus groups We conducted four structured working sections as focus groups to identify, refine, and prioritize Research Questions on GenAI for SE. Focus groups have been used as a means to gather data in SE research [35, 36, 37, 38] Focus group sessions produce mainly qualitative information about the objects of study. The benefits of focus groups are that they produce insightful information, and the method is fairly inexpensive and fast to perform. This differs from a brainstorming section where participants are selected and navigated by a moderator, following a structured protocol so that the discussions stay focused [36, 39]. Kontio et al. also suggest online focus groups with a lot of advantages, such as group thinking, no travel cost, anonymous contribution, and support for large groups [36]. The value of the method is sensitive to the experience and insight of participants. Hence, we presented the information of the participants in Table 1. Focus groups’ timeline is from April 2023 to September 2023 (as shown in Figure 2). All participants are SE researchers who have experience or interest in the topic, as shown in Table 1. For each focus group, we developed a plan, which included the agenda of the section and a set of exercises for the participants. Each focus group lasted between 2 to 3 hours. To capture data from focus groups, we use moderator’s notes, Miro boards 7, and recording in case of online sessions. The focus of each focus group is different: • Focus group 1 (Exploratory brainstorming): Brainstorming was aimed at exploring ideas for potential opportunities and challenges when adopting GenAI in software development activities. We discussed a question, “Which SE areas will benefit from GenAI tools, such as ChatGPT and Copilot?”. We used SWEBOK areas to initialize the conversations. At the end of the group, 11 categories were created. Each category is assigned a section leader who is a co-author responsible for synthesizing content for the section. • Focus group 2 (Exploratory brainstorming): We discussed “What would be an interesting research topic for GenAI and Software Development and management?”. We initiated the discussion about “What could be interesting Research Questions for an empirical study on GenAI in SE?” Some questions were generated during the working session. • Focus group 3 (Validating brainstorming): prior to the meeting, a list of possible RQs was made. The list included 121 RQs. From this meeting, we aimed to validate the questions, if they are correctly and consistently interpreted, and if they are reasonable and meaningful to all participants of the meeting. Participants had sufficient time to review thoroughly and critically the RQs list. Each person left a comment for every question about their meaningfulness and feasibility. After this step, the RQs were revised and restructured. • Focus group 4 (Validating brainstorming): The discussion was conducted in subgroups, each group focused on one particular SE category. Prior to this session, the question list was finalized. In total, there are 78 RQs in 11 categories. RQs that are consensusly not practically important or meaningful are excluded. We also discussed and ranked RQs according to their novelty and timeliness of each RQs. However, we did not achieve consensus and a complete list of rankings for all RQs. Therefore, we decided not to present the ranking in this work. 3.3. Threats to validity 3.3. Threats to validity According to Kontio et al. [37, 36]threats of the focus groups method in SE studies include group dynamics, social acceptability, hidden agenda, secrecy, and limited comprehension. To minimize the group dynamics threat, i.e., uncontrollable flow of discussion, all sections were coordinated by the main author, strictly following the pre-determined agenda with time control. Social acceptability can influence discussion results, i.e. participants contribute biased opinions because of positive or negative perceptions from others. We introduce clear rules for contribution at the beginning of focus groups. For online sessions, Miro is used so participants can anonymously provide their opinions. As time is relatively limited for an online focus group, some complex issues or points might not be not necessarily understood by all participants (limited comprehension). This threat is mitigated by organizing focus groups following up with each other. Between the sessions, participants were required to do their ”homework” so discussing points could be studied or reflected further beyond the scope of the focus groups. Four focus groups occurring in four months give sufficient time for individual and group reflection on the topic.

Generative Artificial Intelligence for Software Engineering: Research Approach

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012;
(2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden;
(3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland;
(4) Chetan Arora, Monash University, Melbourne, Australia;
(5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy;
(6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria;
(7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy;
(8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy;
(9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy;
(10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland;
(11) Mika Saari, Tampere University, Tampere, Finland;
(12)  Zheying Zhang, Tampere University, Tampere, Finland;
(13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam;
(15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction
Background
Research Approach
Research Agenda
Outlook and Conclusions
References Abstract Context Generative Artificial Intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI’s ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Objective. Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Method. We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for Software Engineering. Results. We identified 78 open Research Questions (RQs) in 11 areas of Software Engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision-making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Conclusions. GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research. Keywords: Generative Artificial Intelligence, GenAI, ChatGPT, CoPilot, Software Engineering, Software Project, Software Development, Focus Group, Literature review, Structured review, Literature survey, Research Agenda, Research Roadmap 1. Introduction In recent years, Generative Artificial Intelligence (GenAI) has attracted significant attention and interest from Software Engineering (SE) research and practice. GenAI tools such as GitHub Copilot1 and ChatGPT2 have been rapidly explored in various professional contexts given their remarkable performance in producing human-like content. The emerging adoption of  these tools reintroduces a host of classic concerns about productivity and quality when adopting new technologies. Clearly, code generation and test case optimization are SE tasks that directly benefit from recent large language models (LLMs) [1, 2, 3, 4]. Beyond traditional research on applied. Machine Learning (ML), GenAI tools introduced usability and accessibility, leveraging AI-generated content to a broader range of professionals, requiring less technical competence to work and integrate them into existing work environment. GenAI tools are also showing their potential in non-coding tasks, such as assisting requirements engineering, software processes and project management. At the moment, GenAI is an active research areas with several challenges and open questions. To do well on specific tasks, LLMs require fine-tuning or training. Research has much focused so far on achieving reliable and scalable GenAI output for different SE tasks. GenAI models are inherently nondeterministic: the same prompt produces different answers on different inference executions [5]. Moreover, GenAI’s output can be very sensitive to the input parameters or settings [6, 7]. Hallucination is another common concern with AI-generated content, as they can be imaginary or fictional [8]. The promise of AI automation might be closer than ever in SE [9], when these open challenges are sufficiently addressed. Research agenda is a popular type of work for guiding and organizing research efforts in certain research areas [10, 11, 12, 13, 14, 15]. It often includes a review of existing work, and a presentation of directions, visions and priorities, enabling researchers to make meaningful contributions to their respective fields and to address pressing challenges. This research agenda is driven by past and current work on GenAI for SE. Based on focus groups and a literature review, we portray the current research on GenAI in SE and present the research agenda with open challenges for future research. While we do not emphasize the comprehensiveness of the literature review due to the fast-moving nature of the topic, the focus groups present practical expectations of the future roles of GenAI in software development. The research agenda is organized into 11 areas of concern, which are: (1) Requirements Engineering, (2) Software Design,(3) Sofware Implementation, (4) Quality Assurance, (5) Software Maintenance and Evolution, (6) Software Processes and Tools, (7) Engineering Management, (8) Professional Competencies, (9) Software Engineering Education, (10) Macro Aspects and (11) Fundamental concerns of GenAI. Two international events have provided a ground for this work. As a part of the first international workshop on AI-assisted Agile Software Development3 , we ran our first focus group to explore the benefits and challenges of AI-assisted Agile software development. At the Requirements Engineering (RE) conference, we organized the second international workshop on Requirements Engineering for Software Startups and Emerging Technologies (RESET) 4 with a special theme on GenAI and Requirements Engineering. Consequently, the identified open challenges reflect not just the existing gaps in literature but also draw from the practical, value-driven insights of the co-authors. The paper is structured as follows: we clarify the meaning of GenAI and what we know about GenAI in SE in Section 2, and we will present our research approach in Section 3. Section 4 presents our Research Agenda. Finally, Section 5 is Outlook and Conclusions. This paper is available on arxiv under CC 4.0 license. Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Anh Nguyen-Duc, University of South Eastern Norway, BøI Telemark, Norway3800 and Norwegian University of Science and Technology, Trondheim, Norway7012; (2) Beatriz Cabrero-Daniel, University of Gothenburg, Gothenburg, Sweden; (3) Adam Przybylek, Gdansk University of Technology, Gdansk, Poland; (4) Chetan Arora, Monash University, Melbourne, Australia; (5) Dron Khanna, Free University of Bozen-Bolzano, Bolzano, Italy; (6) Tomas Herda, Austrian Post - Konzern IT, Vienna, Austria; (7) Usman Rafiq, Free University of Bozen-Bolzano, Bolzano, Italy; (8) Jorge Melegati, Free University of Bozen-Bolzano, Bolzano, Italy; (9) Eduardo Guerra, Free University of Bozen-Bolzano, Bolzano, Italy; (10) Kai-Kristian Kemell, University of Helsinki, Helsinki, Finland; (11) Mika Saari, Tampere University, Tampere, Finland; (12)  Zheying Zhang, Tampere University, Tampere, Finland; (13) Huy Le, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (14) Tho Quan, Vietnam National University Ho Chi Minh City, Hochiminh City, Vietnam and Ho Chi Minh City University of Technology, Hochiminh City, Vietnam; (15) Pekka Abrahamsson, Tampere University, Tampere, Finland. Table of Links Abstract and Introduction Background Research Approach Research Agenda Outlook and Conclusions References Abstract and Introduction Abstract and Introduction Background Background Research Approach Research Approach Research Agenda Research Agenda Outlook and Conclusions Outlook and Conclusions References References Abstract Context Generative Artificial Intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI’s ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Context Objective. Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Objective. Method . We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for Software Engineering. Method Results . We identified 78 open Research Questions (RQs) in 11 areas of Software Engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision-making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Results Conclusions. GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research. Conclusions. Keywords: Generative Artificial Intelligence, GenAI, ChatGPT, CoPilot, Software Engineering, Software Project, Software Development, Focus Group, Literature review, Structured review, Literature survey, Research Agenda, Research Roadmap Keywords: 1. Introduction In recent years, Generative Artificial Intelligence (GenAI) has attracted significant attention and interest from Software Engineering (SE) research and practice. GenAI tools such as GitHub Copilot1 and ChatGPT2 have been rapidly explored in various professional contexts given their remarkable performance in producing human-like content. The emerging adoption of  these tools reintroduces a host of classic concerns about productivity and quality when adopting new technologies. Clearly, code generation and test case optimization are SE tasks that directly benefit from recent large language models (LLMs) [1, 2, 3, 4]. Beyond traditional research on applied. Machine Learning (ML), GenAI tools introduced usability and accessibility, leveraging AI-generated content to a broader range of professionals, requiring less technical competence to work and integrate them into existing work environment. GenAI tools are also showing their potential in non-coding tasks, such as assisting requirements engineering, software processes and project management. At the moment, GenAI is an active research areas with several challenges and open questions. To do well on specific tasks, LLMs require fine-tuning or training. Research has much focused so far on achieving reliable and scalable GenAI output for different SE tasks. GenAI models are inherently nondeterministic: the same prompt produces different answers on different inference executions [5]. Moreover, GenAI’s output can be very sensitive to the input parameters or settings [6, 7]. Hallucination is another common concern with AI-generated content, as they can be imaginary or fictional [8]. The promise of AI automation might be closer than ever in SE [9], when these open challenges are sufficiently addressed. Research agenda is a popular type of work for guiding and organizing research efforts in certain research areas [10, 11, 12, 13, 14, 15]. It often includes a review of existing work, and a presentation of directions, visions and priorities, enabling researchers to make meaningful contributions to their respective fields and to address pressing challenges. This research agenda is driven by past and current work on GenAI for SE. Based on focus groups and a literature review, we portray the current research on GenAI in SE and present the research agenda with open challenges for future research. While we do not emphasize the comprehensiveness of the literature review due to the fast-moving nature of the topic, the focus groups present practical expectations of the future roles of GenAI in software development. The research agenda is organized into 11 areas of concern, which are: (1) Requirements Engineering, (2) Software Design,(3) Sofware Implementation, (4) Quality Assurance, (5) Software Maintenance and Evolution, (6) Software Processes and Tools, (7) Engineering Management, (8) Professional Competencies, (9) Software Engineering Education, (10) Macro Aspects and (11) Fundamental concerns of GenAI. Two international events have provided a ground for this work. As a part of the first international workshop on AI-assisted Agile Software Development3 , we ran our first focus group to explore the benefits and challenges of AI-assisted Agile software development. At the Requirements Engineering (RE) conference, we organized the second international workshop on Requirements Engineering for Software Startups and Emerging Technologies (RESET) 4 with a special theme on GenAI and Requirements Engineering. Consequently, the identified open challenges reflect not just the existing gaps in literature but also draw from the practical, value-driven insights of the co-authors. The paper is structured as follows: we clarify the meaning of GenAI and what we know about GenAI in SE in Section 2, and we will present our research approach in Section 3. Section 4 presents our Research Agenda. Finally, Section 5 is Outlook and Conclusions.

Generative Artificial Intelligence for Software Engineering: Abstract and Introduction

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 8 CONCLUSION Automated testing is crucial for helping improve app quality. Despite the dozens of mobile app GUI testing techniques, how to automatically generate the diversified unusual text inputs for fully testing mobile apps remains a challenge. This paper proposes InputBlaster which leverages the LLM to produce the unusual inputs together with the mutation rules which serve as the reasoning chains. It formulates the unusual inputs generation problem as a task of producing a set of test generators, each of which can yield a batch of unusual text inputs under the same mutation rule. The evaluation is conducted for both effectiveness and usefulness, with 136% higher bug detection rate than the best baselines, and uncovering 37 new crashes. In the future, we plan to further analyze the root causes and repair strategy of these input-related bugs, and design automated bug repair methods. REFERENCES [1] 2022. Crash bug text. https://www.theguardian.com/technology/iphone-crashbug-text imessage-ios. [2] 2022. Crash bug text in ios. https://tech.hindustantimes.com/tech/news/becareful-a-new-text bomb-is-making-whatsapp-crash-and-will-hang-yourphone-71599532897852.html. [3] 2022. F-droid. https://f-droid.org/. [4] Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training LLMs for project-specific code-summarization. ASE (2022). [5] Nadia Alshahwan and Mark Harman. 2011. Automated web application testing using search based software engineering. In ASE. IEEE, 3–12. [6] Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated concolic testing of smartphone apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. 1–11. [7] Yauhen Leanidavich Arnatovich, Minh Ngoc Ngo, Tan Hee Beng Kuan, and Charlie Soh. 2016. Achieving high code coverage in android ui testing via automated widget exercising. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC). IEEE, 193–200. [8] Yauhen Leanidavich Arnatovich, Lipo Wang, Ngoc Minh Ngo, and Charlie Soh. 2018. Mobolic: An automated approach to exercising mobile application GUIs using symbiosis of online testing technique and customated input generation. Software: Practice and Experience 48, 5 (2018), 1107 1142. [9] Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications. 641–660. [10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901. [12] Tianqin Cai, Zhao Zhang, and Ping Yang. 2020. Fastbot: A Multi-Agent ModelBased Test Generation System Beijing Bytedance Network Technology Co., Ltd.. In Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test. 93–96. [13] Chen Chen, Baojiang Cui, Jinxin Ma, Runpu Wu, Jianchao Guo, and Wenqian Liu. 2018. A systematic review of fuzzing techniques. Computers & Security 75 (2018), 118–137. [14] Taolue Chen, Alejandro Flores-Lamas, Matthew Hague, Zhilei Han, Denghang Hu, Shuanglong Kan, Anthony W Lin, Philipp Rümmer, and Zhilin Wu. 2022. Solving string constraints with Regex dependent functions through transducers with priorities and variables. Proceedings of the ACM on Programming Languages 6, POPL (2022), 1–31. [15] Taolue Chen, Matthew Hague, Jinlong He, Denghang Hu, Anthony Widjaja Lin, Philipp Rümmer, and Zhilin Wu. 2020. A decision procedure for path feasibility of string manipulating programs with integer data type. In Automated Technology for Verification and Analysis: 18th International Symposium, ATVA 2020, Hanoi, Vietnam, October 19–23, 2020, Proceedings. Springer, 325–342. [16] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. 2020. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems 33 (2020), 22243–22255. [17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). [18] Joel D Day, Thorsten Ehlers, Mitja Kulczynski, Florin Manea, Dirk Nowotka, and Danny Bøgsted Poulsen. 2019. On solving word equations using SAT. In Reachability Problems: 13th International Conference, RP 2019, Brussels, Belgium, September 11–13, 2019, Proceedings 13. Springer, 93–106. [19] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan,  Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In UIST. [20] Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023. Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPT. ISSTA (2023). [21] Yinlin Deng, Chenyuan Yang, Anjiang Wei, and Lingming Zhang. 2022. Fuzzing deep-learning libraries via automated relational API inference. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 44–56. [22] Android Developers. 2012. Ui/application exerciser monkey. [23] Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of android apps. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 481–492. [24] Robert Feldt and Simon Poulding. 2013. Finding test data with specific properties via metaheuristic search. In 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 350–359. [25] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems 35 (2022), 30583–30598. [26] Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 269–280. [27] Yuyu He, Lei Zhang, Zhemin Yang, Yinzhi Cao, Keke Lian, Shuai Li, Wei Yang, Zhibo Zhang, Min Yang, Yuan Zhang, et al. 2020. TextExerciser: feedback-driven text input exercising for android applications. In 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 1071–1087. [28] Luk Holk, Petr Jank, Anthony W Lin, and Rmmer. 2017. String constraints with concatenation and transducers solved efficiently. Proceedings of the ACM on Programming Languages 2, POPL (2017), 1–32. [29] Yang Hu, Umair Z Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. 2019. Re factoring based program repair applied to programming assignments. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 388–398. [30] Text input. 2022. Introduction about text input on Android Developer website. https://developer.android.google.cn/reference/android/widget/EditText?hl=en. [31] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of Code Language Models on Automated Program Repair. ICSE (2023). [32] Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. ICSE (2023). [33] Adam Kiezun, Vijay Ganesh, Shay Artzi, Philip J Guo, Pieter Hooimeijer, and Michael D Ernst. 2013. HAMPI: A solver for word equations over strings, regular expressions, and context-free grammars. ACM Transactions on Software Engineering and Methodology (TOSEM) 21, 4 (2013), 1–28. [34] Sebastian Krings, Joshua Schmidt, Patrick Skowronek, Jannik Dunkelau, and Dierk Ehmke. 2020. Towards constraint logic programming over strings for test data generation. In Declarative Programming and Knowledge Management: Conference on Declarative Programming, DECLARE 2019, Unifying INAP, WLP, and WFLP, Cottbus, Germany, September 9–12, 2019, Revised Selected Papers 22. Springer, 139–159. [35] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. 2020. Reinforcement learning with augmented data. Advances in neural information processing systems 33 (2020), 19884–19895. [36] Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pretrained Large Language Models. In ICSE. [37] Guodong Li and Indradeep Ghosh. 2013. PASS: String solving with parameterized array and interval automaton. In Hardware and Software: Verification and Testing: 9th International Haifa Verification Conference, HVC 2013, Haifa, Israel, November 5-7, 2013, Proceedings 9. Springer, 15–31. [38] Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. DroidBot: A Lightweight UI-Guided Test Input Generator for Android (ICSE-C ’17). [39] Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 23– 26. [40] Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: a deep learning-based approach to automated black-box Android app testing. In ASE. IEEE, 1070–1073. [41] Hongliang Liang, Xiaoxiao Pei, Xiaodong Jia, Wuwei Shen, and Jian Zhang. 2018. Fuzzing: State of the art. IEEE Transactions on Reliability 67, 3 (2018), 1199–1218. [42] Tianyi Liang, Andrew Reynolds, Cesare Tinelli, Clark Barrett, and Morgan Deters. 2014. A DPLL (T) theory solver for a theory of strings and regular expressions. In Computer Aided Verification: 26th International Conference, CAV 2014, Held as Part of the Vienna Summer of Logic, VSL 2014, Vienna, Austria, July 18-22, 2014. Proceedings 26. Springer, 646–662. [43] Peng Liu, Xiangyu Zhang, Marco Pistoia, Yunhui Zheng, Manoel Marques, and Lingfei Zeng. 2017. Automatic text input generation for mobile testing. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 643–653. [44] Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing. arXiv preprint arXiv:2212.04732 (2022). [45] Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl Eyes: Spotting UI Display Issues via Visual Understanding. In ASE. IEEE. https://doi.org/10.1145/3324884.3416547 [46] Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Nighthawk: Fully Automated Localizing UI Display Issues via Visual Understanding. IEEE Transactions on Software Engineering (2022), 1–16. https: //doi.org/10.1109/TSE.2022.3150876 [47] Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding. 48–55. [48] Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 224–234. [49] Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 94–105. [50] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. Computer Science (2013). [51] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv preprint arXiv:2202.12837 (2022). [52] Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-Based Prompt Selection for Code Related Few-Shot Learning. In Proceedings of the 45th International Conference on Software Engineering (ICSE’23). [53] Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153–164. [54] Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. ICLR (2022). [55] Simon Poulding and Robert Feldt. 2017. Generating controllably invalid and atypical inputs for robustness testing. In 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 81–84. [56] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020), 140:1–140:67. [57] Vaibhav Rastogi, Yan Chen, and William Enck. 2013. Appsplayground: automatic security analysis of smartphone applications. In Proceedings of the third ACM conference on Data and application security and privacy. 209–220. [58] J Schulman, B Zoph, C Kim, J Hilton, J Menick, J Weng, JFC Uribe, L Fedus, L Metz, M Pokorny, et al. 2022. ChatGPT: Optimizing language models for dialogue. [59] Carolyn B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Transactions on software engineering 25, 4 (1999), 557–572. [60] Mike Sharples. 2022. Automated Essay Writing: An AIED Opinion. International Journal of Artificial Intelligence in Education (2022), 1–8. [61] Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 245–256. [62] Nezih Sunman, Yiğit Soydan, and Hasan Sözer. 2022. Automated web application testing driven by pre-recorded test cases. Journal of Systems and Software (2022), 111441. [63] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2014. S3: A symbolic string solver for vulnerability detection in web applications. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. 1232–1243. [64] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2017. Model counting for recursively-defined strings. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part II 30. Springer, 399–418. [65] UIAutomator. 2021. Python wrapper of Android uiautomator test tool. https://github. com/xiaocong/uiautomator. [66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems (2017). [67] Jue Wang, Yanyan Jiang, Chang Xu, Chun Cao, Xiaoxing Ma, and Jian Lu. 2020. Combodroid: generating high-quality test inputs for android apps via use case combinations. In ICSE. 469–480. [68] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledgebased vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 3081–3089. [69] Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. 2022. An extensive study on pre-trained models for program understanding and generation. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 39–51. [70] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui  Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022).

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Conclusion and References

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 7 RELATED WORK There have been many automated GUI testing techniques for mobile apps [7, 9, 12, 22, 23, 26, 39, 48, 49, 57, 61, 67], yet they mainly focus on how to plan the exploration paths to fully cover the app activities and states. There are also studies [27, 43, 44] that aim at generating valid inputs to pass the GUI pages and are used to enrich the automated testing tools for higher coverage. None of them can conduct the testing of text input widgets. Testing Related with Text Inputs. For Web apps, SWAT [5] and AWET [62] generated the unusual inputs based on the pre-defined template. ACTEve [6] and S3 [63] first used symbolic execution to extract input constraints in the source code and then employ a solver to generate the inputs. They need to analyze the web code and can’t be directly applied to Android apps which have quite different rendering mechanisms. In addition, some constraints are dynamically generated (as shown in Section 2.1.2), and couldn’t be extracted from the source code. There are some string analysis methods for generating the strings that violate the constraints (e.g., string length) [14, 15, 18, 28, 33, 34, 37, 42, 64]. Although they are effective for string constraints, yet the inputs of mobile apps are more diversified, and they cannot work well in our task. With the breakthrough of LLMs, studies have proposed to explore how LLMs can be used to assist developers in a variety of tasks, such as code generation [54, 69], program repair [29, 31, 52], and code summarization [4, 69]. There is also a growing trend of applying LLM for software testing, e.g., fuzzing deep learning libraries [20], unit test generation [36], bug reproduction [32], valid input generation [44], etc, and achieves significant performance improvement. This work explores a different task, i.e., unusual text input generation for mobile apps, which provides new insights into how LLM can enhance the software testing practice. LLM for Software Engineering.

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Related Work

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 6 DISCUSSION AND THREATS TO VALIDITY 6.1 Generality Across Platforms The primary idea of InputBlaster is to generate unusual inputs for text widgets with the context information when running the apps. Although we only experiment with Android mobile apps, since other platforms have these similar types of information, InputBlaster can be used to conduct the testing of input widgets for other platforms. We conduct a small-scale experiment for another two popular platforms, and experiment on 10 iOS apps with 15 bugs and 10 Web apps with 18 bugs, with details on our website. Results show that InputBlaster’s bug detection rate is 80% for iOS apps and 78% for Web apps within 30 minutes testing time. This further demonstrates the generality and usefulness of InputBlaster, and we will conduct more thorough experiments in the future. 6.2 Threats of Validity The first threat concerns the representativeness of the experimental apps. We have selected popular and active apps which can partially reduce this threat. The second threat relates to the baseline selection. Since there are hardly any existing approaches for the unusual input generation of mobile apps, we employ 18 approaches from various aspects for a thorough comparison. There are inputs generation techniques for Web apps [5, 6, 62, 63], yet because they need to analyze the web code which is different from mobile apps considering the different rendering mechanism, and cannot be directly applied in our task, hence we don’t include them as the baselines. The third threat is that we only focus on the crash bugs, since they cause more serious effects and can be automatically observed, and existing studies also only explore this type of bug [38, 40, 53]. The fourth threat might lie in the process of manual categorization in Section 2.1.2. The process involves multiple practitioners and double-checking for the final decision. Also note that, the derived categorization is only for illustration, rather than serving as the ground truth for evaluation. The Fifth threat may exist in the uncertainty of LLM output results. LLM may not generate the corresponding output as expected, and we also design in-context learning and feedback mechanisms to ensure the output format and content of LLM. Last but not least, InputBlaster gradually builds the example dataset (Section 3.3.1) as the test goes on. This indicates the performance can be influenced by the testing order, e.g., when arranged in the first place, the crash could not be detected, yet when arranged after 10 apps are tested, the crash can be revealed, since the example dataset has accumulated more knowledge. In this paper, we use a random order of the experimental apps and would explore more in the future.

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Discussion and Validity

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 5 RESULTS AND ANALYSIS 5.1 Bugs Detection Performance (RQ1) Table 2 presents the bug detection performance of InputBlaster. With the unusual inputs generated by InputBlaster, the bug detection rate is 0.78 (within 30 minutes), indicating 78 (28/36) of the bugs can be detected. In addition, the bugs can be detected with an average of 13.52 attempts, and the average bug detection time is 9.64 minutes, which is acceptable. This indicates the effectiveness of our approach in generating unusual inputs for testing the app, and facilitating the uncovering of bugs related to input widgets. Figure 5 demonstrates examples of InputBlaster’s generated unusual inputs and the inputs that truly trigger the crash. We can see that our proposed approach can generate quite diversified inputs which mutate the valid input from different aspects, e.g., for the price in the first example which should be a non-negative value, the generated unusual inputs range from negative values and decimals to various kinds of character strings. Furthermore, it is good at capturing the contextual semantic information of the input widgets and their associated constraints, and generating the violations accordingly. For example, for the minimum and maximum price in the first example, it generates the unusual inputs with the minimum larger than the maximum, and successfully triggers the crash. We further analyze the bugs that could not be detected by our approach. A common feature is that they need to be triggered under specific settings, e.g., only under the user-defined setting, the input can trigger the crash, in the environment we tested, it may not have been possible to trigger a crash due to the lack of user-defined settings in advance. We have manually compared the unusual inputs generated by our approach with the ones in the issue reports. We find in all cases, InputBlaster can generate the satisfied buggy inputs within 30 attempts and 30 minutes, which further indicates its effectiveness. Table 2 also shows the performance comparison with the baselines. We can see that our proposed InputBlaster is much better than the baselines, i.e., 136% (0.78 vs. 0.33) higher in bug detection rate (within 30 minutes) compared with the best baseline Text Exerciser. This further indicates the advantages of our approach. Nevertheless, the Text Exerciser can only utilize the dynamic hints in input generation which covers a small portion of all situations, i.e., a large number of input widgets do not involve such feedback. Performance comparison with baselines. Without our elaborate design, the raw ChatGPT demonstrates poor performance, which further indicates the necessity of our approach. In addition, the string analysis methods, which are designed specifically for string constraints, would fail to work for mobile apps. In addition, since the input widgets of mobile apps are more diversified (as shown in Section 2.1.2) compared with the string, the heuristic analysis or finite-state automata techniques in the string analysis methods might be ineffective for our task. The baselines for automated GUI testing or valid text input generation are even worse, since their main focus is to increase the coverage through generating valid inputs. This further implies the value of our approach for targeting this unexplored task. 5.2 Ablation Study (RQ2) Table 3 shows the performance of InputBlaster and its 2 variants respectively removing the first and third module. In detail, for InputBlaster w/o valid Input (i.e., without Module 1), we provide the information related to the input widgets (as Table 1 P1) to the LLM in Module 2 and set other information from Module 1 as “null”. For InputBlaster w/o enrich Examples (i.e., without Module 3), we set the examples from Module 3 as “null” when querying the LLM. Note that, since Module 2 is for generating the unusual inputs which is indispensable for this task, hence we do not experiment with this variant. 5.2.1 Contribution of Modules. We can see that InputBlaster’s bug detection performance is much higher than all other variants, indicating the necessity of the designed modules and the advantage of our approach. Compared with InputBlaster, InputBlaster w/o validInput results in the largest performance decline, i.e., 50% drop (0.39 vs. 0.78) in bug detection rate within 30 minutes. This further indicates that the generated valid inputs and inferred constraints in Module 1 can help LLM understand what the correct input looks like and generate the violated ones. also undergoes a big performance decrease, i.e., 32% (0.53 vs. 0.78) in bug detection rate within 30 minutes, and the average testing time increases by 109% (9.64 vs. 20.15). This might be because without the examples, the LLM would spend more time understanding user intention and criteria for what kinds of answers are wanted. InputBlaster w/o enrichExamples Table 4 further demonstrates the performance of InputBlaster and its 5 variants. We remove each sub-module of the InputBlaster in Figure 3 separately, 5.2.2 Contribution of Sub-modules. i.e., inferred constraint, mutation rule, text execution feedback, test generator and retrieved examples of buggy input. For removing the test generator, we directly let the LLM generate the unusual inputs, and for removing retrieved examples, we use the random selection method. For other variants, we set the removed content as “null”. The experimental results demonstrate that removing any of the sub-modules would result in a noticeable performance decline, indicating the necessity and effectiveness of the designed sub modules. Removing the mutation rules (InputBlaster w/o-mutateRule) have the greatest impact on the performance, reducing the bug detection rate by 50% (0.36 vs. 0.72 within 30 attempts). Remember that, InputBlaster first lets the LLM to generate the mutation rules (how to mutate the valid inputs), then asks it to produce the test generator following the mutation rule. With the generated mutation rules serving as the reasoning chain, the unusual input generation can be more effective, which further proves the usefulness of our design. We also notice that, when removing the test generator (InputBlaster w/o-generator), the bug detection rate does not drop much (0.72 vs. 0.61) when considering 30 attempts, yet it declines a lot (0.78 vs. 0.36) when considering 30 minutes of testing time. This is because our proposed approach lets the LLM produce the test generator which can yield a batch of unusual inputs. This means interacting with the LLM once can generate multiple outcomes. However, if asking the LLM to directly generates unusual inputs (i.e., InputBlaster w/o-generator), it requires interacting with LLM frequently, and could be quite inefficient. This further demonstrates we formulate the problem as producing the test generator task is efficient and valuable. In addition, randomly selecting the examples (InputBlaster w/oretriExample) would also largely influence the performance, and decrease the bug detection rate by 22% (0.56 vs. 0.72 within 30 attempts). This indicates that by providing similar examples, the LLM can quickly think out what should the unusual inputs look like. Nevertheless, we can see that, compared with the variant without enriched examples in prompt (Table 3), the randomly selected examples do take effect (0.47 vs 0.56 in bug detection rate within 30 attempts), which further indicates the demonstration can facilitate the LLM in producing the required output. Table 5 demonstrates the performance under the different number of examples provided in the prompt. 5.2.3 Influence of Different Number of Examples. We can see that the number of detected bugs increases with more examples, reaching the highest bug detection rate with 5 examples. And after that, the performance would gradually decrease even increasing the examples. This indicates that too few or too many examples would both damage the performance, because of the tiny information or the noise in the provided examples. 5.3 Usefulness Evaluation (RQ3) Table 6 shows all bugs spotted by Ape integrated with our InputBlaster, and more detailed information on detected bugs can be seen in our website. For the 131 apps, InputBlaster detects 43 bugs in 32 apps, of which 37 are newly-detected bugs. Furthermore, these new bugs are not detected by the Ape without InputBlaster. We submit these 37 bugs to the development team, and 28 of them have been fixed/confirmed so far (21 fixed and 7 confirmed), while the remaining are still pending (none of them is rejected). This further indicates the effectiveness and usefulness of our proposed InputBlaster in bug detection. When confirming and fixing the bugs, some Android app developers express thanks such as “Very nice! You find an invalid input we thought was too insignificant to cause crashes.”(i.e., Ipsos). Furthermore, some developers also express their thought about the buggy text input “Handling different inputs can be tricky, and I admit we couldn’t test for every possible scenario. It has given me a fresh appreciation for the complexity of user inputs and the potential bugs they can introduce. ”(i.e., DRBUs). Some developers also present valuable suggestions to facilitate the further improvement of InputBlaster. For example, some of them hope that we can find the patterns of these bugs and design repair methods.

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Results and Analysis

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 4 EXPERIMENT DESIGN 4.1 Research Questions • How effective of InputBlaster in detecting bugs related to text input widgets? RQ1: (Bugs Detection Performance) For RQ1, we first present some general views of InputBlaster for bug detection, and then compare it with commonly-used and state-of-the-art baseline approaches. • What is the contribution of the (sub-) modules of InputBlaster for bug detection performance? RQ2: (Ablation Study) For RQ2, We conduct ablation experiments to evaluate the impact of each (sub-) module on the performance. • How does our proposed InputBlaster work in real-world situations? RQ3: (Usefulness Evaluation) For RQ3, we integrate InputBlaster with the GUI testing tool to make it automatically explore the app and detect unseen inputrelated bugs, and issue the detected bugs to the development team. 4.2 Experimental Setup we crawl 200 most popular open-source apps from F-Droid [3], and only keep the latest ones with at least one update after September 2022 (this ensures the utilized apps are not overlapped with the ones in Sec 3.3). Then we collect all their issue reports on GitHub, and use keywords (e.g., EditText) to filter those related to text input. Finally, we obtain 126 issue reports related to 54 apps. Then we manually review each issue report and the mobile app, and filter it according to the following criteria: (1) the app wouldn’t constantly crash on the emulator; (2) it can run all baselines; (3) UIAutomator [65] can obtain the view hierarchy file for context extraction; (4) the bug is related to text input widgets; (5) the bug can be manually reproduced for validation; (6) the app is not used in the motivational study or example dataset construction. Please note that we follow the name of the app to ensure that there is no overlap between the datasets. Finally, 31 apps with 36 buggy text inputs remain for further experiments. For RQ1 and RQ2, We measure the bug detection rate, i.e., the ratio of successfully triggered crashes in terms of all the experimental crashes (i.e., buggy inputs), which is a widely used metric for evaluating GUI testing [8, 27, 43]. Specifically, with the generated unusual input, we design an automated test script to input it into the text input widgets, and automatically run the “submit” operation to check whether a crash occurs. If no, use the script to go back to GUI page with the input widget if necessary, and try the next generated unusual input. As long as a crash is triggered for a text input widget, we treat it as successful bug detection and will stop the generation for this widget. Note that our generated unusual input is not necessarily the same as the one provided in the issue report, e.g., -18 vs. -20, as long as a crash is triggered after entering the unusual inputs, we treat it as a successful crash detection. For a fair comparison with other approaches, we employ two experimental settings, i.e., 30 attempts (30 unusual inputs) and 30 minutes. We record the bug detection rate under each setting (denoted as “Bug (%)” in Table 2 to Table 5), and also record the actual number of attempts (denoted as “Attempt (#)”) and the actual running time (denoted as “Min (#)”) when the crash occurs to fully understanding the performance. we further evaluate the usefulness of InputBlaster in detecting unseen crash bugs related to text input. A total of 131 apps have been retained. We run Ape [26] (a commonly-use automated GUI testing tool) integrated with InputBlaster, for exploring the mobile apps and getting the view hierarchy file of each GUI page. For RQ3, We use the same configurations as the previous experiments. Once a crash related to text input is spotted, we create an issue report by describing the bug, and report them to the app development team through the issue reporting system or email. 4.3 Baselines Since there are hardly any existing approaches for the unusual input generation of mobile apps, we employ 18 baselines from various aspects to provide a thorough comparison. First, we directly utilize ChatGPT [58] as the baseline. We provide the context information of the text input widgets (as described in Table 1 P1), and ask it to generate inputs that can make app crash. Fuzzing testing and mutation testing can be promising techniques for generating invalid inputs, and we apply several related baselines. Feldt et al. [24] proposed a testing framework called GoldTest, which generates diverse test inputs for mobile apps by designing regular expressions and generation strategies. In 2017, they further proposed an invalid input generation method [55] based on probability distribution (PD) parameters and regular expressions, and we name this baseline as PDinvalid. Furthermore, we reuse the idea of traditional random-based fuzzing [13, 41], and develop a RandomFuzz for generating inputs for text widgets. In addition, based on the 50 buggy text inputs from the GitHub dataset in Section 3.3.1, we manually design 50 corresponding mutation rules to generate the invalid input, and name this baseline as ruleMutator. Furthermore, we include the string analysis methods as the baselines, i.e., OSTRICH [15] and Sloth [14]. They aim at generating the strings that violate the constraints (e.g., string length, concatenation, etc), which is similar to our task. OSTRICH’s key idea [15] is to generate the test strings based on heuristic rules. Sloth [14] proposes to exploit succinct alternating finite-state automata as concise symbolic representations of string constraints. There are constraint-based methods, i.e., Mobolic [8] and TextExerciser [27], which can generate diversified inputs for testing the app. For example, TextExerciser utilizes the dynamic hints to guide it in producing the inputs. We also employ two methods (RNNInput [43] and QTypist [44]) which aim at generating valid inputs for passing the GUI page. In addition, we use the automated GUI testing tools, i.e., Stoat [61], Droidbot [39], Ape [26], Fastbot [12], ComboDroid [67], TimeMachine [23], Humanoid [40], Q-testing [53], which can produce inputs randomly or following rules to make app running automatically. We design the script for each baseline to ensure that it can reach the GUI page with the text input widget, and run them in the same experimental environment (Android x64) to mitigate potential bias.

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Experiment Design

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 3 APPROACH This paper aims at automatically generating a batch of unusual text inputs which can possibly make the mobile apps crash. The common practice might directly produce the target inputs with LLM as existing studies in valid input generation [44] and fuzzing deep learning libraries [20, 21]. Yet, this would be quite inefficient for our task, because each interaction with the LLM requires a few seconds waiting for the response and consumes lots of energy. Instead, this paper proposes to produce the test generators (a code snippet) with LLM, each of which can generate a batch of unusual text inputs under the same mutation rule (e.g., insert special characters into a string), as demonstrated in Figure 4 ⑤. To achieve this, we propose InputBlaster which leverages LLM to produce the test generators together with the mutation rules which serve as the reasoning chains for boosting the performance, and each test generator then automatically generates a batch of unusual text inputs, as shown in Figure 3. In detail, given a GUI page with text input widgets and its corresponding view hierarchy file, we first leverage LLM to generate the valid text input which can pass the GUI page (Sec 3.1). We then leverage LLM to produce the test generator which can generate a batch of unusual text inputs, and simultaneously we also ask the LLM to output the mutation rule which serves as the reasoning chain for guiding the LLM in making the effective mutations from valid inputs (Sec 3.2). To further boost the performance, we utilize the in-context learning schema to provide useful examples when querying the LLM, from online issue reports and historical running records (Sec 3.3). 3.1 Prompt Generation for Valid Input InputBlaster first leverages LLM to generate the valid input which will serve as the target towards which the following mutation can be conducted. The context information relates to the input widgets and its belonged GUI page can provide important clues about what the valid input should be, therefore we input this information into LLM (in Section 3.1.1). In addition, we also include the dynamic feedback information when interacting with the input widgets (in Section 3.1.2), and the constraint categories we summarized in the previous section (in Section 3.1.3) to improve the performance. Furthermore, besides the valid text input, we also ask LLM to output its inferred constraints for generating the valid input which will facilitate the approach to generating the mutation rules in the next section. We summarize all the extracted information with examples in Table 1. The context information is extracted from the view hierarchy file, which is easily obtained by automated GUI testing tools [26, 48, 49, 61]. As shown in Table 1, we extract the text-related field of the input widget which indicates how the valid input should be. In detail, we extract the “hint text”, “resource id”, and ‘text’ fields of the input widget, and utilize the first non-empty one among the above three fields. 3.1.1 Context Extraction. We also extract the activity name of the GUI page and the mobile app name, and this global context further helps refine the understanding of the input widget. In addition, we extract the local context of the input widget (i.e., from nearby widgets) to provide thorough viewpoints and help clarify the meaning of the widget. The candidate information source includes the parent node widgets, the leaf node widget, widgets in the same horizontal axis, and fragment of the current GUI page. For each information source, we extract the “text” field (if it is empty, use the “resource-id” field), and concatenate them into the natural-language description with the separator (‘;’). When one inputs an incorrect text into the app, there are some feedbacks (i.e., dynamic hints) related to the inputs, e.g., the app may alter the users that the password should contain letters and digits. The dynamic hint can further help LLM understand what the valid input should look like. 3.1.2 Dynamic Hint Extraction. We extract the dynamic hints via differential analysis which compares the differences of the GUI page before and after inputting the text, and extracts the text field of the newly emerged widgets (e.g., a popup window) in the later GUI page, with examples shown in Figure 2. We also record the text input which makes the dynamic hint happens, which can help the LLM to understand the reason behind it. Our pilot study in Section 2.1.2 summarizes the categories of constraints within and among the widgets. The information can provide direct guidance for the LLM in generating the valid inputs, for example, the constraint explicitly requires the input should be pure text (without special characters). We provide this list of all candidate constraints described in natural language as in Section 2.1.2 to the LLM. 3.1.3 Candidate Constraints Preparation. With the extracted information, we use three kinds of information to generate prompts for inputting into the LLM, as shown in Table 1. Generally speaking, it first provides the context information and the dynamic hints (if any) of the input widgets, followed by the candidate constraints, and then queries the LLM for the valid input. Due to the robustness of LLM, the generated prompt sentence does not need to fully follow the grammar. 3.1.4 Prompt Generation. After inputting the prompt, the LLM will return its recommended valid text input and its inferred constraints, as demonstrated in Figure 4 ②. We then input it into the widget, and check whether it can make the app transfer to the new GUI page (i.e., valid input). If the app fails to transfer, we iterate the process until the valid input is generated. 3.2 Prompt Generation for Test Generator with Mutation Rule Based on the valid input in the previous section, InputBlaster then leverages LLM to produce the test generator together with the mutation rule. As demonstrated in Figure 4 ⑤, the test generator is a code snippet that can generate a batch of unusual inputs, while the mutation rule is the natural language described operation for mutating the valid inputs which automatically output by LLM based on our prompt and serves as the reasoning chain for producing the test generator. Note that the mutation rule here is output by LLM. Each time when a test generator is produced, we can obtain a batch of automatically generated unusual text inputs, and will input them into the text widgets to check whether they have successfully made the mobile app crash. This test execution feedback (in Section 3.2.2) will be incorporated in the prompt for querying the LLM which can enable it more familiar with how the mutation works and potentially produce more diversified outcomes. We also include the inferred constraints in the previous section in the prompt (in Section 3.2.1), since the natural language described explanation would facilitate the LLM in producing effective mutation rules, for example, the inferred constraint is that the input should be in pure text (without special characters) and the LLM would try to insert certain characters to violate the constraint. We have obtained the inferred constraints and valid input from the output of the LLM in the previous section, here we extract this information from the output message and will input it into the LLM in this section. We design a flexible keyword matching method to automatically extract the description between the terms like ‘constraints’ and ‘the input’ and treat it as the inferred constraints, and extract the description after the terms like ‘input is’ and treat it as the valid input, as demonstrated in Figure 4 ②. 3.2.1 Inferred Constraints and Valid Input Extraction. After generating the unusual text inputs, we input them into the mobile app and check whether they can successfully trigger the app crash. This test execution information will be inputted into the LLM to generate more effective and diversified text inputs. We use the real buggy text inputs and the other unusual inputs (which don’t trigger bugs) to prompt LLM in the follow-up generation. The former can remind the LLM to avoid generating duplicate ones, while the latter aims at telling the LLM to consider other mutation rules. 3.2.2 Test Execution Feedback Extraction. Besides, we also associate the mutation rules with the text input to enable the LLM to better capture its semantic meaning. As shown in Figure 4 ⑤, we extract the content between the keywords “Mutation rule” and “Test generator” as mutation rules. With the extracted information, we design linguistic patterns of the prompt for generating the test generator and mutation rules. As shown in Figure 4 ④, the prompt includes four kinds of information, namely inferred constraints, valid input, text execution feedback, and question. The first three kinds of information are mainly based on the extracted information as described above, and we also add some background illustrations to let the LLM better understand the task, like the inferred constraint in Figure 4 ④. For the question, we first ask the LLM to generate the mutation rule for the valid input, then let it produce a test generator following the mutation rule. Due to the robustness of LLM, the generated prompt sentence does not need to follow the grammar completely. 3.2.3 Prompt Generation. 3.3 Enriching Prompt with Examples It is usually difficult for LLM to perform well on domain-specific tasks as ours, and a common practice would be employing the in-context learning schema to boost the performance. It provides the LLM with examples to demonstrate what the instruction is, which enables the LLM better understand the task. Following the schema, along with the prompt for the test generator as described in Section 3.2, we additionally provide the LLM with examples of the unusual inputs. To achieve this, we first build a basic example dataset of buggy inputs (which truly trigger the crash) from the issue reports of open-source mobile apps, and continuously enlarge it with the running records during the testing process (in Section 3.3.1). Based on the example dataset, we design a retrieval-based example selection method (in Section 3.3.2) to choose the most suitable examples in terms of an input widget, which further enables the LLM to learn with pertinence. We collect the buggy text inputs from GitHub and continuously build an example dataset that serves as the basis for in-context learning. For each data instance, as demonstrated in 4 ③, it records the buggy text inputs and the mutation rules which facilitate the LLM understanding of how the buggy inputs come from. It also includes the context information of the input widgets which provides the background information of the buggy inputs, and enables us to select the most suitable examples when querying the LLM. 3.3.1 Example Dataset Construction. First, we automatically crawl the issue reports and pull  requests from the Android mobile apps in GitHub (updated before September 2022). Then we use keyword matching to filter these related to the text inputs (e.g., EditText) and have triggered crashes. Following that, we then employ manual checking to further determine whether there is a crash triggered by the buggy text inputs by running the app. In this way, we obtain 50 unusual inputs and store them in the example dataset (There is no overlap with the evaluation datasets.). We then extract the context information of the input widget with the method in Section 3.1.1, and store it together with the unusual input. Note that, since these buggy inputs don’t associate with the mutation rules, we set them as null. Mining buggy text inputs from GitHub. We enrich the example dataset with the newly emerged unusual text inputs which truly trigger bugs during InputBlaster runs on Enlarging the dataset with buggy text inputs during testing. various apps. Specifically, for each generated unusual text input, after running it in the mobile apps, we put the ones which trigger crashes into the example dataset. We also add their associated mutation rules generated by the LLM, as well as the context information extracted in Section 3.1.1. Examples can provide intuitive guidance to the LLM in accomplishing a task, yet excessive examples might mislead the LLM and cause the performance to decline. Therefore, we design a retrieval based example selection method to choose the most suitable examples (i.e., most similar to the input widgets) for LLM. 3.3.2 Retrieval-based Example Selection and In-context Learning. In detail, the similarity comparison is based on the context information of the input widgets. We use Word2Vec (Lightweight word embedding method) [50] to encode the context information of each input widget into a 300-dimensional sentence embedding, and calculate the cosine similarity between the input widget and each data instance in the example dataset. We choose the top-K data instance with the highest similarity score, and set K as 5 empirically. The selected data instances (i.e., examples) will be provided to the LLM in the format of context information, mutation rule, and buggy text input, as demonstrated in Figure 4 ③. 3.4 Implementation We implement InputBlaster based on the ChatGPT which is released on the OpenAI website[3]. It obtains the view hierarchy file of the current GUI page through UIAutomator [65] to extract context information of the input widgets. InputBlaster can be integrated by replacing the text input generation module of the automated GUI testing tool, which automatically extracts the context information and generates the unusual inputs. [3] https://beta.openai.com/docs/models/chatgpt

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Approach

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References 2 MOTIVATIONAL STUDY AND BACKGROUND To better understand the constraints of text inputs in real-world mobile apps, we carry out a pilot study to examine their prevalence. We also categorize the constraints, to facilitate understanding and the design of our approach for generating unusual inputs violating the constraints. 2.1 Motivational Study The dataset is collected from one of the largest Android GUI datasets Rico [19], which has a great number of Android GUI screenshots and their corresponding view hierarchy files [45, 46]. These apps belong to diversified categories such as news, entertainment, medical, etc. We analyze the view hierarchy file according to the package name and extract the GUI page belonging to the same app. A total of 7,136 apps with each having more than 3 GUI pages are extracted. For these apps, we first randomly select 136 apps with 506 GUI pages and check their text inputs through view hierarchy files. We summarize a set of keywords that indicate the apps have text inputs widgets [30], e.g., EditText, hint-text, AutoCompleteTextView, etc. We then use these keywords to automatically filter the view hierarchy files from the remaining 7,000 apps, and obtain 5,761 candidate apps with at least one potential text input widget. Four authors then manually check them to ensure that they have text inputs until a consensus is reached. In this way, we finally obtain 5,013 (70.2%) apps with at least one text input widget, and there are 3,723 (52.2%) apps having two or more text input widgets. Please note that there is no overlap with the evaluation dataset. 2.1.1 Data Collection. We randomly select 2000 apps with text inputs and conduct manual categorization to derive the constraint types of input widgets. Following the open coding protocol [59], two authors individually examine the content of the text input, including the app name, activity name, input type and input content. Then each annotator iteratively merges similar codes, and any disagreement of the categorization will be handed over to the third experienced researcher for double checking. Finally, we come out with a categorization of the constraints within (intra-widget) and among the widgets (inter-widget), with details summarized in Figure 2. 2.1.2 The Constraint Categories of Text Inputs. Intra-widget constraints depict the requirements of a single text input, e.g., a widget for a human’s height can only input the non-negative number. There are explicit and implicit sub-types. The former accounts for 63%, which manifests as the requirement to display input directly on the GUI page. And the latter account for 37%, mainly manifested as the feedback when incorrect text input is received, e.g., after inputting a simple password, the app would remind the user “at least one upper case character (A-Z) is required” as demonstrated in Figure 2. Intra-widget constraint. Inter-widget constraints depict the requirements among multiple text input widgets on a GUI page, for example, the diastolic pressure should be less than systolic pressure as shown in Figure 2. Inter-widget constraint. As demonstrated above, the text input widgets are quite common in mobile apps, e.g., 70.2% apps with at least one such widget. Furthermore, considering the diversity of inputs and contexts, it would require significant efforts to manually build a complete set of mutation rules to fully test an input widget, and the automated technique is highly demanded. This confirms the popularity of text inputs in mobile apps and the complexity of it for full testing, which motivates us to automatically generate a batch of unusual text inputs for effective testing and bug detection. Summary. 2.2 Background of LLM and In-context Learning The target of this work is to generate the input text, and the Large Language Model (LLM) trained on ultra-large-scale corpus can understand the input prompts (sentences with prepending instructions or a few examples) and generate reasonable text. When pre-trained on billions of samples from the Internet, recent LLMs (like ChatGPT [58], GPT-3 [10] and T5 [56]) encode enough information to support many natural language processing tasks [47, 60, 68]. Tuning a large pre-trained model can be expensive and impractical for researchers, especially when limited fine-tuned data is available for certain tasks. In-context Learning (ICL) [11, 25, 51] offers a new alternative that uses Large Language Models to perform downstream tasks without requiring parameter updates. It leverages input-output demonstration in the prompt to help the model learn the semantics of the task. This new paradigm has achieved impressive results in various tasks, including code generation and assertion generation.

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Study and Background

This paper is under CC 4.0 license. available on arxiv Authors: (1) Zhe Liu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (2) Chunyang Chen, Monash University, Melbourne, Australia; (3) Junjie Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author; (4) Mengzhuo Chen, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (5) Boyu Wu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (6) Zhilin Tian, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (7) Yuekai Huang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (8) Jun Hu, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China; (9) Qing Wang, State Key Laboratory of Intelligent Game, Beijing, China Institute of Software Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China & Corresponding author. Table of Links Abstract and Introduction Motivational Study and Background Approach Experiment Design Results and Analysis Discussion and Threats to Validity Related Work Conclusion and References ABSTRACT Mobile applications have become a ubiquitous part of our daily life, providing users with access to various services and utilities. Text input, as an important interaction channel between users and applications, plays an important role in core functionality such as search queries, authentication, messaging, etc. However, certain special text (e.g., -18 for Font Size) can cause the app to crash, and generating diversified unusual inputs for fully testing the app is highly demanded. Nevertheless, this is also challenging due to the combination of explosion dilemma, high context sensitivity, and complex constraint relations. This paper proposes InputBlaster which leverages the LLM to automatically generate unusual text inputs for mobile app crash detection. It formulates the unusual inputs generation problem as a task of producing a set of test generators, each of which can yield a batch of unusual text inputs under the same mutation rule. In detail, InputBlaster leverages LLM to produce the test generators together with the mutation rules serving as the reasoning chain, and utilizes the in-context learning schema to demonstrate the LLM with examples for boosting the performance. InputBlaster is evaluated on 36 text input widgets with cash bugs involving 31 popular Android apps, and results show that it achieves 78% bug detection rate, with 136% higher than the best baseline. Besides, we integrate it with the automated GUI testing tool and detect 37 unseen crashes in real-world apps from Google Play. KEYWORDS Android GUI testing, Large language model, In-context learning 1 INTRODUCTION Mobile applications (apps) have become an indispensable component of our daily lives, enabling instant access to a myriad of services, information, and communication platforms. The increasing reliance on these applications necessitates a high standard of quality and performance to ensure user satisfaction and maintain a competitive edge in the fast-paced digital landscape. The ubiquity of mobile applications has led to a constant need for rigorous testing and validation to ensure their reliability and resilience against unexpected user inputs. Text input plays a crucial role in the usability and functionality of mobile applications, serving as a primary means for users to interact with and navigate these digital environments [43, 44]. From search queries and form submissions to instant messaging and content creation, text input is integral to the core functionality of numerous mobile applications across various domains. The seamless handling of text input is essential for delivering a positive user experience, as it directly impacts the ease of use, efficiency, and overall satisfaction of the users. Given the unexpected input, the program might suffer from memory leakage, data corruption, falling into the dead loop, resulting in the application stuck, crash, or other serious issues [14, 27, 28, 63]. Even worse, these buggy texts can only demonstrate a tiny difference from the normal text, or they themselves are normal text in other contexts, which makes the issue easily occur and difficult to spot. There has been a fair amount in the news about the crash of iOS and Android systems caused by a special text input [1], which has greatly affected people’s daily lives. For example, in July 2020, a specific character of the Indian language caused iOS devices constantly crash. It has affected a wide range of iOS applications, including iMessage, WhatsApp, and Facebook Messenger [2], and as long as certain text inputs contain the character, these apps would crash. Taken in this sense, automatically generating unusual inputs for fully testing the input widgets and uncovering bugs is highly demanded. Existing automated GUI testing techniques focus on generating the valid text input for passing the GUI page and conducting the follow-up page exploration [6, 8, 27, 43, 44, 62, 63], e.g., QTypist [44] used GPT-3 to generate semantic input text to improve the coverage of the test. They could not be easily adapted to this task, since the unusual inputs can be more diversified and follow different rationales from the valid inputs. There are also studies targeting at generating strings that violate the constraints (e.g., string length) with heuristic analysis or finite state automaton techniques [37, 42, 64]. Yet they are designed for specific string functions like concatenation and replacement, and could not be generalized in this task. Nevertheless, it is very challenging for the automatic generation of diversified unusual inputs. The first challenge is the combination explosion. There can be numerous input formats including text, number, date, time, currency, and innumerable settings, e.g., different character sets, languages and text lengths, which makes it quite difficult if not impossible to enumerate all these variants. The second challenge is context sensitivity. The unusual inputs should closely relate to the context of the input widgets to effectively trigger the bug, e.g., a negative value for font size (as shown in Figure 1), an extremely large number to potentially violate the widget for people’s height. The third challenge is the constraint relation within and among the input widgets. The constraints can be that a widget only accepts pure numbers (without characters), or the sum of item values smaller/bigger than the total (as shown in Figure 1), which requires an exact understanding of the related widgets and these constraints so as to generate targeted variation. What’s more difficult is that certain constraints only appear when interacting with the apps (i.e., dynamic hints in terms of the incorrect texts), and static analysis cannot capture these circumstances. Large Language Models (LLMs) [10, 17, 58, 66, 70] trained on ultra-large-scale corpus have exhibited promising performance in a wide range of tasks. ChatGPT[58], developed by OpenAI, is one such LLM with an impressive 175 billion parameters, trained on a vast dataset. Its ability to comprehend and generate text across various domains is a testament to the potential of LLMs in interacting with humans as knowledgeable experts. The success of ChatGPT is a clear indication that LLMs can understand human knowledge and can do well in providing answers to various questions. Inspired by the fact that the LLM has made outstanding progress in email reply, abstract extraction, etc. [10, 16, 35, 68], we propose an approach, InputBlaster[1] , to automatically generate the unusual text inputs with LLM which uncover the bugs[2] related to the text input widgets. Instead of directly generating the unusual inputs by LLM which is of low efficiency, we formulate the unusual inputs generation problem as a task of producing a set of test generators (a code snippet), each of which can yield a batch of unusual text inputs under the same mutation rule (i.e., insert special characters into a string), as demonstrated in Figure 4 ⑤. To achieve this, InputBlaster leverages LLM to produce the test generators together with the mutation rules which serve as the reasoning chains for boosting the performance. In detail, InputBlaster first leverages LLM to generate the valid input which can pass the GUI page and serves as the target for the follow-up mutation (Module 1). Based on it, it then leverages LLM to produce mutation rules, and asks the LLM to follow those mutation rules and produce the test generator, each of which can yield a batch of unusual text inputs (Module 2). To further boost the performance, we utilize the in-context learning schema to demonstrate the LLM with useful examples from online issue reports and historical running records (Module 3). To evaluate the effectiveness of InputBlaster, we carry out experiments on 36 text input widgets with cash bugs involving 31 popular Android apps in Google Play. Compared with 18 common-used and state-of-the-art baselines, InputBlaster can achieve more than 136% boost in bug detection rate compared with the best baseline, resulting in 78% bugs being detected. In order to further understand the role of each module and sub-module of the approach, we conduct ablation experiments to further demonstrate its effectiveness. We also evaluate the usefulness of InputBlaster by integrating it with the automated GUI testing tool and detecting unseen crash bugs in real-world apps from Google Play. Among 131 apps, InputBlaster detects 37 new crash bugs with 28 of them being confirmed and fixed by developers, while the remaining are still pending. The contributions of this paper are as follows: • We are the first to propose a novel LLM-based approach InputBlaster for the automatic generation of unusual text inputs for mobile app testing. • We conduct the first empirical categorization of the constraint relationships within and among text input widgets, which provides clues for the LLM in effective mutation, and facilitates the follow-up studies on this task. • We carry out the effectiveness and usefulness evaluation of InputBlaster, with a promising performance largely outperforming baselines and 37 new detected bugs. [1] Our approach is named as InputBlaster considering it likes a blaster which ignites the following production of the unusual inputs. [2] Note that, like existing studies [38, 40, 53], this paper focuses on the crash bug, which usually causes more serious effects and can be automatically observed, and we interchangeably use the term bug and crash.

Leveraging LLMs for Generation of Unusual Text Inputs in Mobile App Tests: Abstract and Introduction

Authors:
(1) Vinod S. Khandkar and Manjesh K. Hanawal, Industrial Engineering and Operations Research Indian Institute of Technology Bombay, Mumbai, India and {vinod.khandkar, mhanawal}@iitb.ac.in. Table of Links Abstract & Introduction Related Work and Background Challenges in TD Detection Measurement Setup Development Case Study : Wehe - TD Detection Tool for Mobile Environment Shortcoming of Wehe on HTTPS Traffic TD Detection of HTTPS Traffic Conclusion & References VII. CONCLUSION Net neutrality violation detection is a need of an hour. As many of the ISPs are also content providers these days, they compete with each other, which can lead to one deliberately discriminating the services of the other to gain market share. However, users should have the freedom to choose services as per their wishes. Our work considered various challenges in the detection of traffic discrimination in HTTPS traffic. As a case study, we validated Wehe, one of the latest tools available to detect traffic differentiation. The described challenges helped us divide the entire tool into multiple interdependent components and validate them independently. Our validation using commercial traffic shaper revealed that traffic in Wehe setup may not mimic the characteristics of HTTPS traffic accessed from the original servers. Hence, middle-boxes may not subject them to intended discrimination. Thus, Wehe may not detect discrimination of HTTPS traffic. Our new method that uses the appropriate SNI parameter value in the initial TLS handshake message overcomes this shortcoming. Hence our work provided a mechanism to detect a wide range of possible discriminations on the Internet. REFERENCES [1] T. Garrett, L. E. Setenareski, L. M. Peres, L. C. E. Bona, and E. P. Duarte, “Monitoring network neutrality: A survey on traffic differentiation detection,” IEEE Communications Surveys Tutorials, vol. 20, no. 3, pp. 2486–2517, 2018. [2] V. Nguyen, D. Mohammed, M. Omar, and P. Dean, “Net neutrality around the globe: A survey,” in 2020 3rd International Conference on Information and Computer Technologies (ICICT), 2020, pp. 480–488. [3] Ramneek, P. Hosein, W. Choi, and W. Seok, “Detecting network neutrality violations through packet loss statistics,” in 2015 17th AsiaPacific Network Operations and Management Symposium (APNOMS), 2015, pp. 404–407. [4] D. Li, F. Tian, M. Zhu, L. Wang, and L. Sun, “A novel framework for analysis of global network neutrality based on packet loss rate,” in 2015 International Conference on Cloud Computing and Big Data (CCBD), 2015, pp. 297–304. [5] M. Dischinger, A. Mislove, A. Haeberlen, and K. P. Gummadi, “Detecting bittorrent blocking,” in Proceedings of the 8th ACM SIGCOMM Conference on Internet Measurement, Oct. 2008, pp. 3–8. [6] M. B. Tariq, M. Motiwala, N. Feamster, and M. Ammar, “Detecting network neutrality violations with causal inference,” in Proceedings of the 5th International Conference on Emerging Networking Experiments and Technologies, ser. CoNEXT ’09. New York, NY, USA: Association for Computing Machinery, 2009, p. 289–300. [Online]. Available: https://doi.org/10.1145/1658939.1658972 [7] R. Ravaioli, G. Urvoy-Keller, and C. Barakat, “Towards a general solution for detecting traffic differentiation at the internet access,” in 2015 27th International Teletraffic Congress, Sep. 2015, pp. 1–9. [8] Y. Zhang, Z. M. Mao, and M. Zhang, “Detecting traffic differentiation in backbone isps with netpolice,” in Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, ser. IMC ’09. New York, NY, USA: ACM, 2009, pp. 103–115. [Online]. Available: http://doi.acm.org/10.1145/1644893.1644905 [9] P. Kanuparthy and C. Dovrolis, “Shaperprobe: End-to-end detection of isp traffic shaping using active methods,” in Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, Nov. 2011, pp. 473–482. [10] U. Weinsberg, A. Soule, and L. Massoulie, “Inferring traffic shaping and policy parameters using end host measurements,” in Proceedings of IEEE INFOCOM, 2011. [11] M. Dischinger, M. Marcon, S. Guha, K. Gummadi, R. Mahajan, and S. Saroiu, “Glasnost: Enabling end users to detect traffic differentiation,” in 7th USENIX Conf. Networked Systems Design and Implementation, Apr. 2010, pp. 27–27. [12] V. Bashko, N. Melnikov, A. Sehgal, and J. Schonw ¨ alder, “Bonafide: A ¨ traffic shaping detection tool for mobile networks,” in 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), 2013, pp. 328–335. [13] A. Molavi Kakhki, A. Razaghpanah, A. Li, H. Koo, R. Golani, D. Choffnes, P. Gill, and A. Mislove, “Identifying traffic differentiation in mobile networks,” in Proceedings of the 2015 Internet Measurement Conference, Oct. 2015, pp. 239–251. [14] R. Narisetty and D. Gurkan, “Identification of network measurement challenges in openflow-based service chaining,” in 39th Annual IEEE Conference on Local Computer Networks Workshops, 2014, pp. 663– 670. [15] E. Balestrieri, L. De Vito, F. Lamonaca, F. Picariello, S. Rapuano, and I. Tudosa, “Research challenges in measurement for internet of things systems,” ACTA IMEKO, vol. 7, p. 82, 01 2019. [16] E. Karoly. (2000, Feb.) Qos performance validation in real life scenarios. [Online]. Available: http://archive.opengroup.org/tech/qos/ conference/q101/karoly.pdf [17] S. Molnar, P. Megyesi, and G. Szab ´ o, “How to validate traffic gen- ´ erators?” in 2013 IEEE International Conference on Communications Workshops (ICC), 2013, pp. 1340–1344. [18] G. Lu, Y. Chen, S. Birrer, F. E. Bustamante, C. Y. Cheung, and X. Li, “End-to-end inference of router packet forwarding priority,” in IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications, 2007, pp. 1784–1792. [19] R. Mahajan, M. Zhang, L. Poole, and V. Pai, “Uncovering performance differences among backbone isps with netdiff,” in Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, 2008, pp. 205–218. [20] (2008, Jun.) Ookla, measuring and understanding broadband : Speed, quality and application. [Online]. Available: https://www.ookla.com/ docs/UnderstandingBroadbandMeasurement.pdf [21] V. S. Khandkar and M. K. Hanawal, “Detection of traffic discrimination in the internet,” in 2020 International Conference on COMmunication Systems NETworkS (COMSNETS), 2020, pp. 677–679. [22] Tcpreplay: Pcap editing and replay tools for *nix. [Online]. Available: https://github.com/appneta/tcpreplay [23] W. Cui, V. Paxson, N. Weaver, and R. Katz, “Protocol-independent adaptive replay of application dialog,” in 13th Annual Network and Distributed System Security Symposium (NDSS), Feb. 2006, pp. 27–27. [24] R. T. El-Maghraby, N. M. Abd Elazim, and A. M. Bahaa-Eldin, “A survey on deep packet inspection,” in 2017 12th International Conference on Computer Engineering and Systems (ICCES), 2017, pp. 188–197. [25] M. AlSabah, K. Bauer, and I. Goldberg, “Enhancing tor’s performance using real-time traffic classification,” in Proceedings of the 2012 ACM Conference on Computer and Communications Security, ser. CCS ’12. New York, NY, USA: Association for Computing Machinery, 2012, p. 73–84. [Online]. Available: https://doi.org/10.1145/2382196.2382208 [26] Z. Hu, L. Zhu, J. Heidemann, A. Mankin, D. Wessels, and P. Hoffman, “The Transport Layer Security (TLS) Protocol Version 1.3,” Internet Requests for Comments, IETF, RFC 7858, August 2018. [Online]. Available: https://tools.ietf.org/pdf/rfc8446.pdf [27] C. Zhou, C. Lin, and Z. Guo, “mdash: A markov decision-based rate adaptation approach for dynamic http streaming,” IEEE Transactions on Multimedia, vol. 18, no. 4, pp. 738–751, Apr. 2016. [28] T. Joe, L. Eliot, M. Allison, K. Markku, O. Kumiko, S. Martin, E. Lars, M. Alexey, E. Wes, Z. Alexander, T. Brian, I. Jana, M. Allison, T. Michael, K. Eddie, and N. Yoshifumi. (2020, Sept) Service name and transport protocol port number registry. [Online]. Available: https://www.iana.org/assignments/ service-names-port-numbers/service-names-port-numbers.xhtml [29] G. Keinan. (2018, January) Optimizing enterprise networks through sd-avc (software define application visibility and control). [Online]. Available: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2018/ pdf/BRKCRS-2502.pdf [30] A. Rao, J. Sherry, A. Legout, A. Krishnamurthy, W. Dabbous, and D. Choffnes, “Meddle: middleboxes for increased transparency and control of mobile traffic,” in CoNEXT Student Workshop, ACM, 12 2012, pp. 65–66. [31] R. Liu and X. Yu, “A survey on encrypted traffic identification,” in Proceedings of the 2020 International Conference on Cyberspace Innovation of Advanced Technologies, ser. CIAT 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 159–163. [Online]. Available: https://doi.org/10.1145/3444370.3444564 [32] F. Li, A. A. Niaki, D. Choffnes, P. Gill, and A. Mislove, “A large-scale analysis of deployed traffic differentiation practices,” in Proceedings of the ACM Special Interest Group on Data Communication, ser. SIGCOMM ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 130–144. [Online]. Available: https://doi.org/10.1145/3341302.3342092 [33] P. Srisuresh and K. Egevang, “Traditional IP Network Address Translator (Traditional NAT),” Internet Requests for Comments, IETF, RFC 3022, Jan 2001. [Online]. Available: https://tools.ietf.org/pdf/rfc3022.pdf This paper is available on arxiv under CC 4.0 license. Authors: (1) Vinod S. Khandkar and Manjesh K. Hanawal, Industrial Engineering and Operations Research Indian Institute of Technology Bombay, Mumbai, India and {vinod.khandkar, mhanawal}@iitb.ac.in. Authors: Authors: (1) Vinod S. Khandkar and Manjesh K. Hanawal, Industrial Engineering and Operations Research Indian Institute of Technology Bombay, Mumbai, India and {vinod.khandkar, mhanawal}@iitb.ac.in. Table of Links Abstract & Introduction Abstract & Introduction Related Work and Background Related Work and Background Challenges in TD Detection Measurement Setup Development Challenges in TD Detection Measurement Setup Development Case Study : Wehe - TD Detection Tool for Mobile Environment Case Study : Wehe - TD Detection Tool for Mobile Environment Shortcoming of Wehe on HTTPS Traffic Shortcoming of Wehe on HTTPS Traffic TD Detection of HTTPS Traffic TD Detection of HTTPS Traffic Conclusion & References Conclusion & References VII. CONCLUSION Net neutrality violation detection is a need of an hour. As many of the ISPs are also content providers these days, they compete with each other, which can lead to one deliberately discriminating the services of the other to gain market share. However, users should have the freedom to choose services as per their wishes. Our work considered various challenges in the detection of traffic discrimination in HTTPS traffic. As a case study, we validated Wehe, one of the latest tools available to detect traffic differentiation. The described challenges helped us divide the entire tool into multiple interdependent components and validate them independently. Our validation using commercial traffic shaper revealed that traffic in Wehe setup may not mimic the characteristics of HTTPS traffic accessed from the original servers. Hence, middle-boxes may not subject them to intended discrimination. Thus, Wehe may not detect discrimination of HTTPS traffic. Our new method that uses the appropriate SNI parameter value in the initial TLS handshake message overcomes this shortcoming. Hence our work provided a mechanism to detect a wide range of possible discriminations on the Internet. REFERENCES [1] T. Garrett, L. E. Setenareski, L. M. Peres, L. C. E. Bona, and E. P. Duarte, “Monitoring network neutrality: A survey on traffic differentiation detection,” IEEE Communications Surveys Tutorials, vol. 20, no. 3, pp. 2486–2517, 2018. [2] V. Nguyen, D. Mohammed, M. Omar, and P. Dean, “Net neutrality around the globe: A survey,” in 2020 3rd International Conference on Information and Computer Technologies (ICICT), 2020, pp. 480–488. [3] Ramneek, P. Hosein, W. Choi, and W. Seok, “Detecting network neutrality violations through packet loss statistics,” in 2015 17th AsiaPacific Network Operations and Management Symposium (APNOMS), 2015, pp. 404–407. [4] D. Li, F. Tian, M. Zhu, L. Wang, and L. Sun, “A novel framework for analysis of global network neutrality based on packet loss rate,” in 2015 International Conference on Cloud Computing and Big Data (CCBD), 2015, pp. 297–304. [5] M. Dischinger, A. Mislove, A. Haeberlen, and K. P. Gummadi, “Detecting bittorrent blocking,” in Proceedings of the 8th ACM SIGCOMM Conference on Internet Measurement, Oct. 2008, pp. 3–8. [6] M. B. Tariq, M. Motiwala, N. Feamster, and M. Ammar, “Detecting network neutrality violations with causal inference,” in Proceedings of the 5th International Conference on Emerging Networking Experiments and Technologies, ser. CoNEXT ’09. New York, NY, USA: Association for Computing Machinery, 2009, p. 289–300. [Online]. Available: https://doi.org/10.1145/1658939.1658972 [7] R. Ravaioli, G. Urvoy-Keller, and C. Barakat, “Towards a general solution for detecting traffic differentiation at the internet access,” in 2015 27th International Teletraffic Congress, Sep. 2015, pp. 1–9. [8] Y. Zhang, Z. M. Mao, and M. Zhang, “Detecting traffic differentiation in backbone isps with netpolice,” in Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, ser. IMC ’09. New York, NY, USA: ACM, 2009, pp. 103–115. [Online]. Available: http://doi.acm.org/10.1145/1644893.1644905 [9] P. Kanuparthy and C. Dovrolis, “Shaperprobe: End-to-end detection of isp traffic shaping using active methods,” in Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference, Nov. 2011, pp. 473–482. [10] U. Weinsberg, A. Soule, and L. Massoulie, “Inferring traffic shaping and policy parameters using end host measurements,” in Proceedings of IEEE INFOCOM, 2011. [11] M. Dischinger, M. Marcon, S. Guha, K. Gummadi, R. Mahajan, and S. Saroiu, “Glasnost: Enabling end users to detect traffic differentiation,” in 7th USENIX Conf. Networked Systems Design and Implementation, Apr. 2010, pp. 27–27. [12] V. Bashko, N. Melnikov, A. Sehgal, and J. Schonw ¨ alder, “Bonafide: A ¨ traffic shaping detection tool for mobile networks,” in 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), 2013, pp. 328–335. [13] A. Molavi Kakhki, A. Razaghpanah, A. Li, H. Koo, R. Golani, D. Choffnes, P. Gill, and A. Mislove, “Identifying traffic differentiation in mobile networks,” in Proceedings of the 2015 Internet Measurement Conference, Oct. 2015, pp. 239–251. [14] R. Narisetty and D. Gurkan, “Identification of network measurement challenges in openflow-based service chaining,” in 39th Annual IEEE Conference on Local Computer Networks Workshops, 2014, pp. 663– 670. [15] E. Balestrieri, L. De Vito, F. Lamonaca, F. Picariello, S. Rapuano, and I. Tudosa, “Research challenges in measurement for internet of things systems,” ACTA IMEKO, vol. 7, p. 82, 01 2019. [16] E. Karoly. (2000, Feb.) Qos performance validation in real life scenarios. [Online]. Available: http://archive.opengroup.org/tech/qos/ conference/q101/karoly.pdf [17] S. Molnar, P. Megyesi, and G. Szab ´ o, “How to validate traffic gen- ´ erators?” in 2013 IEEE International Conference on Communications Workshops (ICC), 2013, pp. 1340–1344. [18] G. Lu, Y. Chen, S. Birrer, F. E. Bustamante, C. Y. Cheung, and X. Li, “End-to-end inference of router packet forwarding priority,” in IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications, 2007, pp. 1784–1792. [19] R. Mahajan, M. Zhang, L. Poole, and V. Pai, “Uncovering performance differences among backbone isps with netdiff,” in Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, 2008, pp. 205–218. [20] (2008, Jun.) Ookla, measuring and understanding broadband : Speed, quality and application. [Online]. Available: https://www.ookla.com/ docs/UnderstandingBroadbandMeasurement.pdf [21] V. S. Khandkar and M. K. Hanawal, “Detection of traffic discrimination in the internet,” in 2020 International Conference on COMmunication Systems NETworkS (COMSNETS), 2020, pp. 677–679. [22] Tcpreplay: Pcap editing and replay tools for *nix. [Online]. Available: https://github.com/appneta/tcpreplay [23] W. Cui, V. Paxson, N. Weaver, and R. Katz, “Protocol-independent adaptive replay of application dialog,” in 13th Annual Network and Distributed System Security Symposium (NDSS), Feb. 2006, pp. 27–27. [24] R. T. El-Maghraby, N. M. Abd Elazim, and A. M. Bahaa-Eldin, “A survey on deep packet inspection,” in 2017 12th International Conference on Computer Engineering and Systems (ICCES), 2017, pp. 188–197. [25] M. AlSabah, K. Bauer, and I. Goldberg, “Enhancing tor’s performance using real-time traffic classification,” in Proceedings of the 2012 ACM Conference on Computer and Communications Security, ser. CCS ’12. New York, NY, USA: Association for Computing Machinery, 2012, p. 73–84. [Online]. Available: https://doi.org/10.1145/2382196.2382208 [26] Z. Hu, L. Zhu, J. Heidemann, A. Mankin, D. Wessels, and P. Hoffman, “The Transport Layer Security (TLS) Protocol Version 1.3,” Internet Requests for Comments, IETF, RFC 7858, August 2018. [Online]. Available: https://tools.ietf.org/pdf/rfc8446.pdf [27] C. Zhou, C. Lin, and Z. Guo, “mdash: A markov decision-based rate adaptation approach for dynamic http streaming,” IEEE Transactions on Multimedia, vol. 18, no. 4, pp. 738–751, Apr. 2016. [28] T. Joe, L. Eliot, M. Allison, K. Markku, O. Kumiko, S. Martin, E. Lars, M. Alexey, E. Wes, Z. Alexander, T. Brian, I. Jana, M. Allison, T. Michael, K. Eddie, and N. Yoshifumi. (2020, Sept) Service name and transport protocol port number registry. [Online]. Available: https://www.iana.org/assignments/ service-names-port-numbers/service-names-port-numbers.xhtml [29] G. Keinan. (2018, January) Optimizing enterprise networks through sd-avc (software define application visibility and control). [Online]. Available: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2018/ pdf/BRKCRS-2502.pdf [30] A. Rao, J. Sherry, A. Legout, A. Krishnamurthy, W. Dabbous, and D. Choffnes, “Meddle: middleboxes for increased transparency and control of mobile traffic,” in CoNEXT Student Workshop, ACM, 12 2012, pp. 65–66. [31] R. Liu and X. Yu, “A survey on encrypted traffic identification,” in Proceedings of the 2020 International Conference on Cyberspace Innovation of Advanced Technologies, ser. CIAT 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 159–163. [Online]. Available: https://doi.org/10.1145/3444370.3444564 [32] F. Li, A. A. Niaki, D. Choffnes, P. Gill, and A. Mislove, “A large-scale analysis of deployed traffic differentiation practices,” in Proceedings of the ACM Special Interest Group on Data Communication, ser. SIGCOMM ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 130–144. [Online]. Available: https://doi.org/10.1145/3341302.3342092 [33] P. Srisuresh and K. Egevang, “Traditional IP Network Address Translator (Traditional NAT),” Internet Requests for Comments, IETF, RFC 3022, Jan 2001. [Online]. Available: https://tools.ietf.org/pdf/rfc3022.pdf This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Ensuring Fair Internet Access: Challenges and Solutions in Traffic Discrimination Detection

Authors: (1) LI LI, Beihang University, China; (2) XIANG GAO, Beihang University, China; (3) HAILONG SUN, Beihang University, China; (4) CHUNMING HU, Beihang University, China; (5) XIAOYU SUN, The Australian National University, Australia; (6) HAOYU WANG, Huazhong University of Science and Technology, China; (7) HAIPENG CAI, Washington State University, Pullman, USA; (8) TING SU, East China Normal University, China; (9) XIAPU LUO, The Hong Kong Polytechnic University, China; (10) TEGAWENDÉ F. BISSYANDÉ, University of Luxembourg, Luxembourg; (11) JACQUES KLEIN, University of Luxembourg, Luxembourg; (12) JOHN GRUNDY, Monash University, Australia; (13) TAO XIE, Peking University, China; (14) HAIBO CHEN, Shanghai Jiao Tong University, China; (15) HUAIMIN WANG, National University of Defense Technology, China. Table of Links Introduction Background of OpenHarmony The State Of OpenHarmony Ecosystem Overview Of Mobile Software Engineering The Research Roadmap Discussion Related Work Conclusion & References 7 RELATED WORK OpenHarmony software engineering is in its early stage and there are only limited works contributed to this field. Indeed, as highlighted in Section 3.4, there are only 8 papers presented on this aspect. In this section, we will not discuss these OpenHarmony-related works anymore. Instead, we take this opportunity to highlight related works that provide a research roadmap or position statement for guiding a new research field, or a survey including literature reviews for summarizing a mature research direction. We now highlight the representative ones. . One of the most representative research roadmap reports is the one presented by Cheng et al. [22] who have proposed to conduct software engineering research for self-adaptive systems. After thorough discussions among the authors at a Dagstuhl seminar on Software Engineering for Self-Adaptive Systems, the authors have identified four views that are deemed essential to the software engineering of self-adaptive systems. For each view, the authors then summarize the state-of-the-art and highlight the challenges that should be addressed in order to achieve the final goal, i.e., the software is able to automatically cope with the complexity of today’s software-intensive systems. The authors have released another version (called the second research roadmap) five years later after the success of the first version. The goal of this second roadmap paper [25] remains the same, i.e., to summarize the state-of-the-art and to identify critical challenges for the systematic software engineering of self-adaptive systems. Other representative research roadmap papers include the one proposed by France et al. [34] who advocate model-driven development of complex software as well as the one proposed by Papazoglou et al. [85] who advocate service-oriented computing as a new computing paradigm for supporting the development of rapid, low-cost and easy composition of distributed applications. Both of these works have summarized the state-of-the-art and challenges faced by ongoing research activities. More recently, McDermott et al. [79] present a research roadmap about Artificial Intelligence for Software Engineering (AI4SE) and Software Engineering for Artificial Intelligence (SE4AI), presenting key aspects aiming at enabling traditional systems engineering practice automation (AI4SE), and encourage new systems engineering practices supporting a new wave of automated, adaptive, and learning systems (SE4AI). Research Roadmap A literature review involves surveying scholarly sources (mainly research publications) on a specific topic, aiming to provide an overview of the state-of-the-art that is further backed up with a critical evaluation of the material. Except for providing a reflection on the past, it also gives a clear picture of the state of knowledge on the subject that is helpful for guiding future research directions. Because of the aforementioned benefits, in this work, we have resorted to surveying the literature review papers (instead of the majority of primary publications) presented in the field of mobile software engineering. Actually, conducting a survey of surveys is not new to the community. Our fellow researchers have explored this type of study in various domains when the number of primary publications kept increasing until it became difficult to follow the growing body of literature papers in the field. For example, AI-Zewairi et al. [3] have conducted a survey of surveys related to agile software development methodologies, which have gained rigorous attention in the software engineering community with an excessive number of research studies published. As another example, McNabb et al. [80] have presented to the community a survey of surveys about information visualization, which has also become extremely popular and the number of publications has become increasingly difficult to follow. Other representative works include the one proposed by Giraldo et al. [41] who have proposed a survey of surveys on the topic of security and privacy in Cyber-physical systems as well as the one proposed by Chatzimparmpas et al. [20] who have conducted a survey of surveys on the use of visualization for interpreting machine learning models. Literature Review. This paper is under CC 4.0 license. available on arxiv

Software Engineering for OpenHarmony: Related Work

OpenHarmony-এর জন্য সফটওয়্যার ইঞ্জিনিয়ারিং: সম্পর্কিত কাজ

Softwareentwicklung für OpenHarmony: Verwandte Arbeiten

Ingeniería de software para OpenHarmony: trabajos relacionados

Génie logiciel pour OpenHarmony : travaux connexes

ओपनहार्मनी के लिए सॉफ्टवेयर इंजीनियरिंग: संबंधित कार्य

OpenHarmony のソフトウェア エンジニアリング: 関連作業

OpenHarmony를 위한 소프트웨어 엔지니어링: 관련 작업

Engenharia de software para OpenHarmony: trabalhos relacionados

Разработка программного обеспечения для OpenHarmony: сопутствующие работы

OpenHarmony için Yazılım Mühendisliği: İlgili Çalışmalar

Kỹ thuật phần mềm cho OpenHarmony: Công việc liên quan

OpenHarmony 软件工程：相关工作

Software Engineering for OpenHarmony—A Research Roadmap: Conclusion & References

Authors: (1) LI LI, Beihang University, China; (2) XIANG GAO, Beihang University, China; (3) HAILONG SUN, Beihang University, China; (4) CHUNMING HU, Beihang University, China; (5) XIAOYU SUN, The Australian National University, Australia; (6) HAOYU WANG, Huazhong University of Science and Technology, China; (7) HAIPENG CAI, Washington State University, Pullman, USA; (8) TING SU, East China Normal University, China; (9) XIAPU LUO, The Hong Kong Polytechnic University, China; (10) TEGAWENDÉ F. BISSYANDÉ, University of Luxembourg, Luxembourg; (11) JACQUES KLEIN, University of Luxembourg, Luxembourg; (12) JOHN GRUNDY, Monash University, Australia; (13) TAO XIE, Peking University, China; (14) HAIBO CHEN, Shanghai Jiao Tong University, China; (15) HUAIMIN WANG, National University of Defense Technology, China. Table of Links Introduction Background of OpenHarmony The State Of OpenHarmony Ecosystem Overview Of Mobile Software Engineering The Research Roadmap Discussion Related Work Conclusion & References 6 DISCUSSION OpenHarmony, as an emerging mobile platform, is still in its early stage, and so is OpenHarmonyfocused software engineering research. As summarized previously, although there are plenty of opportunities for our fellow researchers to explore in this field, there are still various challenges that need to be addressed. In this section, we highlight some of the representative ones. 6.1 Challenges in App/Library Development. In this work, we have highlighted the gaps that require to be filled in order to catch up with the popular mobile platforms (i.e., Android and iOS). Towards filling the gaps we argue that there are still a number of challenges that need to be addressed. The rise of large language models has been demonstrated to be promising for automated code generation, automated test case generation, library API recommendation, etc. However, it is not yet possible to directly achieve that for OpenHarmony as there is generally no data available for training (or fine-tuning). Even with a set of OpenHarmonyrelated software data (e.g., ArkTS code and its comments), there is also a requirement to further distill high-quality ones in order to achieve a highly precise large language model, as the performance of large language models is known to be highly correlated with the quality of the training dataset. Lacking Data for (AI-based ) Learning. At the moment, there are only a limited number of libraries (in ArkTS) available for supporting the implementation of OpenHarmony apps. The lack of third-party libraries makes it difficult for developers to implement OpenHarmony apps as many of the functions need to be developed from scratch. To fill this gap, the OpenHarmony community is currently encouraging practitioners and researchers to translate popular libraries in other languages to ArkTS. However, this simple translation campaign will introduce another challenge, which is to keep updating the library following the updates of the original version. To that end, we argue that dedicated efforts are required to ensure the maintainability of these libraries. Lacking Third-party Libraries. 6.2 Challenges in App/Library Analysis After app (or library) development, there is a strong need to ensure that the app/library satisfies the requirements and is of high quality. The relevant challenges include the newly designed system architecture of OpenHarmony, the comprehensive GUI interactions, the newly introduced app programming language, etc. We now summarize the representative ones. The Android system has introduced various challenges to the software engineering community in order to develop automated approaches to analyze Android apps. First, Android takes components to construct apps, for which the components by themselves are independently developed. The components will not be directly connected at the code side and the actual invocation (via the so-called Inter-Component Communication (ICC) mechanism) will be done over the system. This ICC mechanism could also be leveraged to implement inter-app communications, making it a challenge to perform inter-app analyses. Second, the components in Android are designed to be run over a set of pre-defined methods (known as lifecycle methods) that will be triggered by the system following a certain order. These lifecycle methods are not connected at the code site as well, making it also a challenge for static app analysis (from the analyzer’s point of view, there is no relationship between two lifecycle methods, despite they may be continuously called by the system). Third, similar to that of lifecycle methods, there are callback methods that are not directly connected to the app code as well. These callback methods are directly invoked by the system when certain events (either system events such as receiving an SMS or UI events such as clicking a button) are triggered. OpenHarmony generally shares the same challenges as that of Android. System-related Challenges. The GUI part has been known to be a challenge for precisely analyzing Android apps. First of all, a given GUI page often contains a comprehensive view tree that includes various widgets with different types positioned via different layout strategies. The widgets in the GUI page are further associated with interactive actions (e.g., a button is associated with a click event). Furthermore, a given GUI page may contain different groups of widgets that will only be rendered if a certain condition is satisfied. In OpenHarmony, the analysis of GUI pages is even more challenging as its design principle encourages to use a single component (i.e., Ability) to implement multiple visual pages, which would be implemented via multiple components (i.e., Activities, one page per Activity) in Android. GUI-related Challenges. The language used to implement mobile apps per se may introduce challenges to the software engineering community. For example, in the Android world, the reflection mechanism (inherited from Java) has been known to be a challenge for static analysis. OpenHarmony takes a new language called ArkTS for developers to implement OpenHarmony apps and the ArkTS language per se may introduce various challenges to the software engineering community as well. Indeed, ArkTS allows defining functions with optional parameters and default parameters, which may cause inconsistency between the function signature and its usage in practice. Language-induced Challenges. This paper is under CC 4.0 license. available on arxiv

Stories, News & Mentions about MODASH

Modash’s Stories on HackerNoon