A Large-Scale Analysis of Inclusiveness-Related User Feedback : Conclusion & References

Table of Links

Abstract & Introduction

Motivation

Related Work

Methodology

A Taxonomy of Inclusiveness

Inclusiveness Concerns in Different Types of Apps

Inclusiveness Across Different Sources of User Feedback

Automated Identification of Inclusiveness User Feedback

Discussion

Conclusion & References

10 CONCLUSION

Our study follows a socio-technical grounded theory approach to gain a deeper understanding of inclusiveness related user feedback from end users. Across manual analysis of over 23K user feedback posts from Reddit, Twitter, and Google Play Store regarding 50 popular for profit apps, we build a taxonomy of inclusiveness. Our taxonomy has six main categories, including fairness, technology, privacy, demography, usability, and other human values. The classifier that we train on our data shows that we can automatically identify inclusiveness related feedback among general user feedback. Our results indicate to practitioners that these online sources contain a rich trove of inclusiveness feedback that organizations should consider to build more inclusive software products for diverse end users. We also present our labelled dataset that researchers can use to refine tooling to better support practitioners.

REFERENCES

[1] A. Savidis and C. Stephanidis, “Inclusive development: Software engineering requirements for universally accessible interactions,” Interacting with Computers, vol. 18, no. 1, pp. 71–116, 2006.

[2] J. Shepherd, “23 essential twitter statistics you need to know in 2023,” https://thesocialshepherd.com/blog/twitterstatistics#: :text=TwitterMay 2023.

[3] T. Gaurdian, “Twitter apologises for ’racist’ image-cropping algorithm,” https://www.theguardian.com/technology/2020/sep/21/twitterapologises-for-racist-image-cropping-algorithm, 2020.

[4] Z. S. Li, M. Sihag, N. N. Arony, J. B. Junior, T. Phan, N. Ernst, and D. Damian, “Narratives: the unforeseen influencer of privacy concerns,” in 2022 IEEE 30th International Requirements Engineering Conference (RE). IEEE, 2022, pp. 127–139.

[5] M. Shahin, M. Zahedi, H. Khalajzadeh, and A. R. Nasab, “A study of gender discussions in mobile apps,” arXiv preprint arXiv:2303.09808, 2023.

[6] A. Alshayban, I. Ahmed, and S. Malek, “Accessibility issues in android apps: state of affairs, sentiments, and ways forward,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1323–1334.

[7] H. Khalajzadeh, M. Shahin, H. O. Obie, P. Agrawal, and J. Grundy, “Supporting developers in addressing human-centric issues in mobile apps,” IEEE Transactions on Software Engineering, 2022.

[8] J. Tizard, T. Rietz, X. Liu, and K. Blincoe, “Voice of the users: an extended study of software feedback engagement,” Requirements Engineering, vol. 27, no. 3, pp. 293–315, 2022.

[9] R. Hoda, “Socio-technical grounded theory for software engineering,” IEEE Transactions on Software Engineering, vol. 48, no. 10, pp. 3808–3832, 2021.

[10] K. B. M. , “Designing inclusive software in Windows - Windows apps,” https://learn.microsoft.com/enus/windows/apps/design/accessibility/designing-inclusivesoftware, may 13 2022.

[11] M. Pattison and A. W. Stedmon, “Inclusive design and human factors: Designing mobile phones for older users.” PsychNology J., vol. 4, no. 3, pp. 267–284, 2006.

[12] D. Metaxa-Kakavouli, K. Wang, J. A. Landay, and J. Hancock, “Gender-inclusive design: Sense of belonging and bias in web interfaces,” in Proceedings of the 2018 CHI Conference on human factors in computing systems, 2018, pp. 1–6.

[13] M. Burnett, S. Stumpf, J. Macbeth, S. Makri, L. Beckwith, I. Kwan, A. Peters, and W. Jernigan, “Gendermag: A method for evaluating software’s gender inclusiveness,” Interacting with Computers, vol. 28, no. 6, pp. 760–787, 2016.

[14] I. Nunes, A. Moreira, and J. Araujo, “Conceptual modeling of gender-inclusive requirements,” in International Conference on Conceptual Modeling. Springer, 2021, pp. 395–409.

[15] ——, “Gire: Gender-inclusive requirements engineering,” Data & Knowledge Engineering, vol. 143, p. 102108, 2023.

[16] M. Vorvoreanu, L. Zhang, Y.-H. Huang, C. Hilderbrand, Z. SteineHanson, and M. Burnett, “From gender biases to gender-inclusive design: An empirical investigation,” in Proceedings of the 2019 CHI Conference on human factors in computing systems, 2019, pp. 1–14.

[17] J. C. Grundy, “Impact of end user human aspects on software engineering.” in ENASE, 2021, pp. 9–20.

[18] J. Grundy, I. Mueller, A. Madugalla, H. Khalajzadeh, H. O. Obie, J. McIntosh, and T. Kanij, “Addressing the influence of end user human aspects on software engineering,” in International Conference on Evaluation of Novel Approaches to Software Engineering. Springer, 2021, pp. 241–264.

[19] S. Donetto, P. Pierri, V. Tsianakas, and G. Robert, “Experiencebased co-design and healthcare improvement: realizing participatory design in the public sector,” The Design Journal, vol. 18, no. 2, pp. 227–248, 2015.

[20] E. C. Groen, J. Doerr, and S. Adam, “Towards crowd-based requirements engineering a research preview,” in Requirements Engineering: Foundation for Software Quality: 21st International Working Conference, REFSQ 2015, Essen, Germany, March 23-26, 2015. Proceedings 21. Springer, 2015, pp. 247–253.

[21] D. Pagano and B. Bruegge, “User involvement in software evolution practice: A case study,” in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 953–962.

[22] E. C. Groen, N. Seyff, R. Ali, F. Dalpiaz, J. Doerr, E. Guzman, M. Hosseini, J. Marco, M. Oriol, A. Perini et al., “The crowd in requirements engineering: The landscape and challenges,” IEEE software, vol. 34, no. 2, pp. 44–52, 2017.

[23] W. Maalej and H. Nabil, “Bug report, feature request, or simply praise? on automatically classifying app reviews,” in 2015 IEEE 23rd international requirements engineering conference (RE). IEEE, 2015, pp. 116–125.

[24] G. Williams and A. Mahmoud, “Mining twitter feeds for software user requirements,” in 2017 IEEE 25th International Requirements Engineering Conference (RE). IEEE, 2017, pp. 1–10.

[25] T. Iqbal, M. Khan, K. Taveter, and N. Seyff, “Mining reddit as a new source for software requirements,” in 2021 IEEE 29th International Requirements Engineering Conference (RE). IEEE, 2021, pp. 128–138.

[26] M. Fazzini, H. Khalajzadeh, O. Haggag, Z. Li, H. Obie, C. Arora, W. Hussain, and J. Grundy, “Characterizing human aspects in reviews of covid-19 apps,” in Proceedings of the 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems, 2022, pp. 38–49.

[27] L. Olson, E. Guzman, and F. Kunneman, “Along the margins: ´ Marginalized communities’ ethical concerns about social platforms,” arXiv preprint arXiv:2304.08882, 2023.

[28] E. Guzman, M. Ibrahim, and M. Glinz, “A little bird told me: Mining tweets for requirements and software evolution,” in 2017 IEEE 25th International Requirements Engineering Conference (RE). IEEE, 2017, pp. 11–20.

[29] Watchful1, “Subreddit comments/submissions 2005- 06 to 2022-12.” [Online]. Available: https: //www.reddit.com/r/pushshift/comments/11ef9if/separate dump files for the top 20k subreddits/

[30] N. N. Arony, Z. S. Li, B. Xu, and D. Damian, “Inclusiveness Matters: A Large-Scale Analysis of User Feedback,” Oct. 2023. [Online]. Available: https://doi.org/10.5281/zenodo.10050673

[31] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 492–28 518.

[32] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pretraining of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.

[33] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019.

[34] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, ´ “Unsupervised cross-lingual representation learning at scale,” arXiv preprint arXiv:1911.02116, 2019.

[35] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019.

[36] M. Sihag, Z. S. Li, A. Dash, N. N. Arony, K. Devathasan, N. Ernst, A. B. Albu, and D. Damian, “A data-driven approach for finding requirements relevant feedback from tiktok and youtube,” in 2023 IEEE 31st International Requirements Engineering Conference (RE). IEEE, 2023, pp. 111–122.

[37] R. Akbani, S. Kwek, and N. Japkowicz, “Applying support vector machines to imbalanced datasets,” in Machine Learning: ECML 2004: 15th European Conference on Machine Learning, Pisa, Italy, September 20-24, 2004. Proceedings 15. Springer, 2004, pp. 39–50.

[38] S. Parashar, S. Dhar, and U. Dhar, “Perception of values: a study of future professionals,” Journal of Human Values, vol. 10, no. 2, pp. 143–152, 2004.

[39] S. H. Schwartz, “Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries,” in Advances in experimental social psychology. Elsevier, 1992, vol. 25, pp. 1–65.

[40] Y. Brun and A. Meliou, “Software fairness,” in Proceedings of the 2018 26th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2018, pp. 754–759.

[41] S. Akter, G. McCarthy, S. Sajib, K. Michael, Y. K. Dwivedi, J. D’Ambra, and K. N. Shen, “Algorithmic bias in data-driven innovation in the age of ai,” p. 102387, 2021.

[43] J. Larson, S. Mattu, L. Kirchner, and J. Angwin, “How We Analyzed the COMPAS Recidivism Algorithm.” [Online]. Available: https://www.propublica.org/article/ how-we-analyzed-the-compas-recidivism-algorithm

[44] A. Tobin and A. Kofman, “Facebook Finally Agrees to Eliminate Tool That Enabled Discriminatory Advertising,” Jun. 2022. [Online]. Available: https://www.propublica.org/article/ facebook-doj-advertising-discrimination-settlement

[45] J. Chakraborty, S. Majumder, and T. Menzies, “Bias in machine learning software: Why? how? what to do?” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 429–440.

[46] S. Krishnan, J. Patel, M. J. Franklin, and K. Goldberg, “A methodology for learning, analyzing, and mitigating social influence bias in recommender systems,” in Proceedings of the 8th ACM Conference on Recommender systems, 2014, pp. 137–144.

[47] V. Tsintzou, E. Pitoura, and P. Tsaparas, “Bias disparity in recommendation systems,” arXiv preprint arXiv:1811.01461, 2018.

[48] G. Hofstede, “Cultures and organizations: Software of the mind (london and new york, mcgraw hill),” House, RJ, Hanges, PJ, Javidan, M., Dorfman, PW, & Gupta, V.(Eds. 2004), Airaksinen, pp. 1–25, 1991.

[49] N. Al-Huwail, S. Al-Sharhan, and A. Al-Hunaiyyan, “Learning design for a successful blended e-learning environment: Cultural dimensions,” INFOCOMP Journal of Computer Science, vol. 6, no. 4, pp. 60–69, 2007.

[50] R. A.-L. Fischer, R. Walczuch, and E. Guzman, “Does culture matter? impact of individualism and uncertainty avoidance on app reviews,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). IEEE, 2021, pp. 67–76.

[51] D. Potnis, “Culture’s consequences: Economic barriers to owning mobile phones experienced by women in india,” Telematics and Informatics, vol. 33, no. 2, pp. 356–369, 2016.

[52] A. De Angeli, U. Athavankar, A. Joshi, L. Coventry, and G. I. Johnson, “Introducing atms in india: a contextual inquiry,” Interacting with Computers, vol. 16, no. 1, pp. 29–44, 2004.

[53] M. R. Roller and P. J. Lavrakas, Applied qualitative research design: A total quality framework approach, ser. Applied qualitative research design: A total quality framework approach. New York, NY, US: The Guilford Press, 2015, pages: xviii, 398.