Existing Literature on Inclusive Software and User Feedback

Table of Links

Abstract & Introduction

Motivation

Related Work

Methodology

A Taxonomy of Inclusiveness

Inclusiveness Concerns in Different Types of Apps

Inclusiveness Across Different Sources of User Feedback

Automated Identification of Inclusiveness User Feedback

Discussion

Conclusion & References

3 RELATED WORK

In this section, we describe the existing literature on inclusive software and user feedback.

3.1 Inclusive Software

The term inclusive software is related to the notion of “universal access” which implies software that is accessible and usable by everyone [1]. The underlying philosophy behind designing an inclusive product is to ensure that the product can be used by as many different users as possible rather than excluding anyone [11]. A study by Savidis and Stephanidis [1] highlights the significance of providing the necessary tools to support inclusive software design and development. They indicate that an important aspect of inclusive software development is identifying user requirements that emerge from interaction with the software.

Conventionally, software is developed with a focus on the average user, and requirements for the software are developed with this perspective. Recent studies, however, highlight the need for more inclusive software. For example, although software is primarily intended to be neutral, software interfaces often contain stereotypical visual components that negatively impact many users’ senses of belonging [12]. Burnett et al. [13] revealed that problem-solving software is developed with a perception that users will adopt the features through tinkering. However, statistically, these features are preferred by men than by women, making the software less inclusive for women.

There are some prior works toward building more inclusive software, particularly focusing on gender inclusion. Nunes et al. [14] proposed a conceptual model for genderinclusive requirements that involves creating a genderinclusive requirements document. The document supports practitioners in integrating the model into the requirement elicitation process. Upon evaluation of the model, they found 83.9% positive response in terms of the usefulness of the model [15]. The GenderMag (Gender Inclusiveness Magnifier) method developed by Burnett et al. [13] uses personas encapsulating five facets of gender differences to analyze gender inclusivity in software. An empirical investigation of GenderMag identified biases in an industrial software and helped derive design changes that improved the inclusiveness of the software [16]. Guizani et al., in their study, proposed a Why/Where/Fix approach to find and fix inclusivity bugs in an Open-Source Software project. They reported their approach reduced inclusivity bugs by 90%.

While the studies focus on addressing gender-related concerns, the concept of inclusion extends beyond gender. Recent studies have indicated that to make software more inclusive, software companies need to better understand human aspects such as age, emotions, personality, human values, gender, ethnicity, and culture [17], [18]. There are various ways to understand the different human aspects of diverse users. For example, co-design or participatory design techniques where users are invited to participate and provide feedback during the design process [19]. However, as software grows and becomes more prevalent around the world, it becomes difficult for companies to conduct such design sessions. In such cases, CrowdRE (Crowd Requirement Engineering) [20] techniques can be leveraged. In our study, we complement current research and use the CrowdRE method of exploring inclusion from an end-user perspective in comparison to prior studies that focused on understanding inclusion from a developer or designer perspective.

3.2 User Feedback

Prior literature has shown that user feedback is beneficial for continuous improvement of software quality [21]. User feedback from the online platforms, i.e., from the “Crowd” [22], has been studied to identify a variety of user needs. One common type of user feedback found in app reviews [23], Twitter (now referred to as X) [24], and Reddit [25] are feature requests and bug reports.

More recently, Fazzi et al. [26] analyzed 2,611 app reviews from 57 COVID-19 apps and found nine categories of human aspect related discussions that impact software usage. The authors implied that these human aspects are not always taken into consideration and should be addressed during development. Another study on 1,500 top free Android apps more focused on accessibility issues revealed that the majority of these apps contain significant problems that prevent individuals with disabilities from using the apps [6]. The study demonstrated that various sub-categories of human aspects are identifiable from user feedback, which can raise awareness amongst developers and companies, enabling them to incorporate these insights during development. Similarly, Shahin et al. [5] conducted an analysis of gender related discussions on app reviews and found six major categories: AppFeatures, Appearance, Content, Company Policy and Censorship, Advertisement, and Community. In addition, they automated the identification of gender and non-gender related discussions and acquired an F1-score of 90.77%. Li et al. [4] obtained 4.5 million posts from Reddit and found 9 significant topics related to privacy concerns. Likewise, Olson et al. [27] examined 586 subreddit communities and identified discussions on ethical concerns from end users regarding social platforms. These studies provide empirical evidence that Reddit is a valuable source for gathering and understanding user concerns.

Khalajzadeh et al. [7] examined (manually) 1,200 app reviews and 1,200 GitHub issue comments for 12 open source projects and characterised human-centric issues into three major categories: App Usage, Inclusiveness, and User Reaction. They present the categories in the form of a taxonomy for human-centric issues and employ machine learning and deep learning models to automatically classify humancentric issues. 31 inclusiveness related posts were identified from the user feedback in Google Play reviews. While insightful, they represent only a starting point – the open-source apps lack the full diversity of software end users and, therefore, are insufficient to build a deeper understanding of diverse end user concerns related to inclusion.

To address the gap, we conduct an analysis of inclusiveness from an end-user perspective and analyze end-user feedback for 50 of the most popular software applications in the world, using three popular sources such as Reddit, Google Play Store, and Twitter. We do not consider GitHub as a source in this study because it primarily represents the developer’s perspective rather than that of the end users. Lastly, in contrast to prior work employing an iterative, open coding analysis method, we employ a socio-technical grounded theory [9] approach in our study.