Table of links Table of links Abstract Abstract Abstract 1 Introduction 1 Introduction 1 Introduction 2 Related Work 2 Related Wor 2 Related Wor k 2.1 Fairness and Bias in Recommendations 2.2 Quantifying Gender Associations in Natural Language Processing Representations 3 Problem Statement 3 Problem Statement 3 Problem Statement 4 Methodology 4 Methodology 4 Methodology 4.1 Scope 4.3 Flag 5 Case Study 5 Case Study 5 Case Study 5.1 Scope 5.2 Implementation 5.3 Flag 6 Results 6 Results 6 Results 6.1 Latent Space Visualizations 6.2 Bias Directions 6.3 Bias Amplification Metrics 6.4 Classification Scenarios 7 Discussion 7 Discussion 7 Discussion 8 Limitations & Future Work 8 Limitations & Future Work 8 Limitations & Future Work 9 Conclusion and References 9 Conclusion and References 9 Conclusion and References 7 Discussion 7 Discussion Our case study applied our proposed framework to observe how attribute association bias changes when removing user gender as a feature in an industry LFR model. Gonen and Goldberg [30] suggested the capability for user gender bias to be systematic bias embedded within the latent space, thus making it difficult for simple mitigation techniques to address the core issue; they noted “a systematic bias found in the embeddings, which is independent of the gender direction.” Given this independence, debiasing methods grounded in removing the gender direction were found to be “superficial” fixes. Systematic bias in our case study, similar to that found in word embeddings, would result in significant attribute association bias even when user gender is not included as a model feature'. Like the results by Gonen and Goldberg [30], our case study suggests that gender stereotypes can become implicitly embedded in the representations of both users and items, as supported by the persistence of this bias after removing user gender as a feature during model training. We found that removing user gender as a feature resulted in a statistically significant decrease in levels of attribute association bias, but significant attribute association bias still remained. The presence of such implicit attribute association bias signals the potential for systematic gender bias when using LFR models and representations, or potentially recommendation algorithms in general, for serving podcasts to users. This finding is not surprising given previous research detailing the highly gendered nature of podcast listening [12, 18, 52]. Our findings based on our framework demonstrate that, similar to Gonen and Goldberg [30], known systematic bias can be found and quantified in recommendations. Given this, it is essential for practitioners to audit for attribute association bias when systematic bias is a known factor in their recommendation scenario, such as podcast recommendations. Additionally, our finding means that more straightforward measures, such as removing gender as a feature during training, can also be seen as a “superficial” attempt to entirely remove gender bias from the representation space. Our results concerning the relatively small changes in our metrics and classification outputs show that relying on feature removal in recommendations will not fully mitigate user gender bias entirely from the user or item vectors. Additionally, the findings demonstrate that implicit attribute association bias can occur in LFR models, signaling that this type of bias may need to be accounted for in representation learning beyond NLP and image processing. In light of this, we did find that removing user gender during training of a deep LFR does reduce the inequality of gender association between male and female users to their stereotypical genres and content. In the case of podcast listening, it is unlikely that gender bias can be removed entirely due to the systematic bias found in listening trends showcased by users and found in external research. However, removing gender as a distinct feature could be a first step in reducing the overall amplification of gendered associations reflected in the relationships between user and item vectors within the trained latent space. Our findings supporting the possibility for systematic bias to occur as attribute association bias in LFR outputs leads us to a more challenging question for the research community: when is it appropriate to mitigate systematic bias? If it is found that implicit attribute association bias improves user experience, how should one reduce the risk of representative harm? In some cases, like ours, stereotyped behavior is typical, and some may argue that it is beneficial for providing valuable content recommendations to users. In the case of user gender bias, the harm lies in the model potentially reinforcing stereotypes by driving users towards gendered listening habits they might otherwise not partake in. It is possible to monitor levels of attribute association bias over time to flag increasing bias in the latent space. Nevertheless, when do levels of reinforcing and implicit bias become harmful? Both the research and practitioner community would benefit from more exploration of how to approach setting baselines for managing representative harms in recommendations. Authors:

Lex Beattie
Isabel Corpus
Lucy H. Lin
Praveen Ravichandran Authors: Lex Beattie
Isabel Corpus
Lucy H. Lin
Praveen Ravichandran Lex Beattie Lex Beattie Isabel Corpus Isabel Corpus Lucy H. Lin Lucy H. Lin Praveen Ravichandran Praveen Ravichandran This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Can We Ever Fully Remove Bias from AI Recommendation Systems?

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Guide to Exploiting Unknown Strategies From a Known Bias Set

AI for All: Operationalizing Diversity and Inclusion Requirements for AI Systems

Defining Diversity and Inclusion in AI

Concrete Steps for Implementing Diversity and Inclusion in AI

Research Methodology for Operationalizing Diversity and Inclusion Requirements for AI Systems

Operationalizing Diversity and Inclusion Requirements for AI Systems: Results

A Guide to Exploiting Unknown Strategies From a Known Bias Set

AI for All: Operationalizing Diversity and Inclusion Requirements for AI Systems

Defining Diversity and Inclusion in AI

Concrete Steps for Implementing Diversity and Inclusion in AI

Research Methodology for Operationalizing Diversity and Inclusion Requirements for AI Systems

Operationalizing Diversity and Inclusion Requirements for AI Systems: Results

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps