AI Bias Isn't Always Loud—Sometimes Its Laissez-Faire

Authors: (1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org); (2) Faye-Marie Vassel, Stanford University; (3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology; (4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Table of Links Abstract and 1 Introduction 1.1 Related Work and Contributions 2 Methods and Data Collection 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.2 Modeling Gender, Sexual Orientation, and Race 3 Analysis 3.1 Harms of Omission 3.2 Harms of Subordination 3.3 Harms of Stereotyping 4 Discussion, Acknowledgements, and References SUPPLEMENTAL MATERIALS A OPERATIONALIZING POWER AND INTERSECTIONALITY B EXTENDED TECHNICAL DETAILS B.1 Modeling Gender and Sexual Orientation B.2 Modeling Race B.3 Automated Data Mining of Textual Cues B.4 Representation Ratio B.5 Subordination Ratio B.6 Median Racialized Subordination Ratio B.7 Extended Cues for Stereotype Analysis B.8 Statistical Methods C ADDITIONAL EXAMPLES C.1 Most Common Names Generated by LM per Race C.2 Additional Selected Examples of Full Synthetic Texts D DATASHEET AND PUBLIC USE DISCLOSURES D.1 Datasheet for Laissez-Faire Prompts Dataset 4 DISCUSSION We demonstrate that LM-generated Laissez-Faire harms of omission, subordination, and stereotyping are widespread and pervasive. These harms affect consumers across races, genders, and sexual orientations, and are present in synthetic text generations spanning educational contexts, workplace settings, and interpersonal relationships. Implicit bias and discrimination continue to be overlooked by model developers in favor of self-audits under the relatively new categories of “AI safety” and “red-teaming”, repurposing terms that originate from fields such as computer security [30]. Such framings give more attention to malicious users, national security concerns, or future existential risks as opposed to threats to fundamental human rights that models intrinsically pose to unsuspecting consumers in everyday interactions [71]. Despite lacking rigorous evidence, developers use terms like “Helpful, Harmless, Honest” or “Responsible” to market their LMs [72, 73]. However, our study finds the opposite for minoritized consumers. We provide evidence that generative language models exacerbate harmful racist and sexist ideologies for everyday consumers with scale and efficiency. The ideological bias we discover is especially pernicious as it does not require explicit prompting from “bad actors” to reinforce the omission and subordination of minoritized groups. This in turn increases the risks of psychological harm via subliminal stereotype priming [49]. Far from harmless, these leading LMs respond to everyday prompts with synthetic text that reinforces hegemonic systems built to maintain status quo power structures and produce Laissez-Faire harms for consumers in the process. Results highlight the triple harms of broad omission in the power-neutral condition and in the power-laden condition, extreme ratios of subordination and prevalent stereotyping. Combined, these outputs contribute to a lived experience where consumers with minoritized identities, if they are to be represented at all, only see themselves portrayed by language models as characters who are “struggling students” (as opposed to “star students”), “patients” or “defendants” (as opposed to “doctors” or “lawyers”), and a friend or romantic partner who is subservient and more likely to borrow money or do the chores for someone else. These subordinated portrayals are up to thousands of times more likely to occur than empowering portrayals (see Fig. 3c), a scale of harms that are not without consequence. Omission, subordination, and stereotyping through racialized and gendered textual cues are shown to have direct consequences on consumer health and psychological well-being. Extreme degrees of subordination are especially consequential given well-established results from social psychology that show how the magnitude and duration of stereotyping harms are proportional to the frequency of linguistic triggers [49]. These result in harms that disproportionately affect minoritized groups – harms that are “not borne by people not stereotyped this way” [40] – resulting in cognitive load leading to significant changes in behavior, self-perception, and even impairments in cognitive performance, shown by other studies of the impacts of consuming biased media [41]. Even for consumers who do not inhabit stereotyped identities, such stereotypes reinforce harmful pre-existing prejudices of other groups [42]. Most concerningly, the prompts we study correspond directly to scenarios where LMs are expanding to have direct and unmediated interactions with vulnerable consumers, from AI-assisted writing for K-12 and university students [8, 12] to text-based bots for simulating romantic and intimate interactions [13, 14]. By releasing these models as general-purpose interfaces, LM developers risk propagating Laissez-Faire harms to an untold number of susceptible secondary consumers who use products built on their models. These include consumers in international contexts, who are not covered by the U.S.-centric focus of this initial study. Our results highlight an urgent need for further research that adapts the framework of Laissez-Faire harms to examine more prompts in additional languages, locales, and power contexts. Such studies may still leverage the framework of intersectionality, replacing U.S.-centric identity categories with power structures specific to international contexts (e.g. using caste instead of race, where appropriate). To aid further research and additional insights that researchers with diverse cultural knowledge and lived experiences may find in our study data, we release our LaissezFaire Prompts Dataset at https://doi.org/10.7910/DVN/WF8PJD and provide additional technical details for readers to reproduce our results in Supplement B. We document our dataset with a Datasheet [79] in Supplement D. Harms in generative, chat-based LMs are much more pervasive than previously described. Our findings are especially urgent given the limited set of regulatory human-rights protections for consumers thus far, underscoring the need for multiple reforms in generative AI policy. First, we advocate for intersectional and sociotechnical approaches towards addressing the structural gaps that have enabled developers to sell recent language models as general-purpose tools to an unregulated number of consumer markets, while also remaining vague about or refusing to define the types of harms that are addressed in their self-audits (see [74, 75] for examples). Second, our findings bolster recent calls for greater transparency from LM developers [76] in terms of providing the public with details of the training datasets, model architectures, and labeling protocols used in the creation of generative LMs, given that each of these steps can contribute to the types of bias we observe in our experiments [34]. Finally, given the disproportionate impacts of such harms on underserved communities, we highlight the urgent need for critical and culturally relevant global AI education and literacy programs to inform, protect, and empower diverse consumers in the face of the various Laissez-Faire harms they may experience as they interface with the current proliferation of generative AI tools. ACKNOWLEDGEMENTS Authors T.M-W. and CRS acknowledge funding support from the National Science Foundation under award number SOS2152288. F-MV acknowledges funding support from the National Science Foundation under award number CCF-1918549. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank Diego Kozlowski, Stella Chen, Rahul Gupta-Iwasaki, Bryan Brown, Jay Kim, Dakota Murray, Zarek Drozda, Ashley Ding, Princewill Okoroafor, Gerald Higginbotham, and Hideo Mabuchi for helpful inputs and discussion. REFERENCES [1] O’Neil, L. These Women Tried to Warn Us About AI. Rolling Stone (2023). Available at: https://www.rollingstone.com/culture/culturefeatures/women-warnings-ai-danger-risk-before-chatgpt-1234804367/ (Accessed: 17th December 2023) [2] Sheng, E., Chang, K.-W., Natarajan, P. & Peng, N. The Woman Worked as a Babysitter: On Biases in Language Generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019). Doi:10.18653/v1/d19-1339 [3] Sweeney, L. Discrimination in Online Ad Delivery. Queue 11, 10–29 (2013). [4] Dhamala, J. Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K. W., & Gupta, R. Bold: Dataset and Metrics for Measuring Biases in OpenEnded Language Generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021). Doi:10.1145/3442188.3445924 [5] Blodgett, S. L., Barocas, S., Daumé III, H. & Wallach, H. Language (technology) IS POWER: A Critical Survey of “Bias” in NLP. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020). Doi:10.18653/v1/2020.acl-main.485 [6] Metz, C. What Exactly Are the Dangers Posed by A.I.? The New York Times (2023). Available at: https://www.nytimes.com/2023/05/01/technology/aiproblems-danger-chatgpt.html (Accessed: 17th December 2023) [7] Nguyen, T., Jump, A. & Casey, D. Emerging Tech Impact Radar: 2023. Gartner (2023). Available at: https://www.gartner.com/en/doc/emergingtechnologies-and-trends-impact-radar-excerpt. (Accessed: 17th December 2023) [8] Extance, A. ChatGPT has entered the classroom: How LLMs could transform education. Nature 623, 474–477 (2023). [9] Sal Khan. 2023. How AI could save (not destroy) education. Sal Khan: How AI could save (not destroy) education, TED Talk (April 2023). [10] Peeples, J. The Future of Education? California Teachers Association (2023). Available at: https://www.cta.org/educator/posts/the-future-of-education. (Accessed: 17th December 2023) [11] OpenAI. Teaching with AI (2023). Available at: https://openai.com/blog/teaching-with-ai. (Accessed: 17th December 2023) [12] Hayden Field. 2024. OpenAI Announces First Partnership With a University. CNBC (2024). Retrieved from: https://www.cnbc.com/2024/01/18/openai-announces-first-partnership-with-a-university.html (Accessed: 19th January 2024). [13] Chow, A. R. Why People Are Confessing Their Love for AI Chatbots. Time (2023). Available at: https://time.com/6257790/ai-chatbots-love/. (Accessed: 17th December 2023) [14] Carballo, R. Using A.I. to Talk to the Dead. The New York Times (2023). Available at: https://www.nytimes.com/2023/12/11/technology/ai-chatbotsdead-relatives.html. (Accessed: 17th December 2023) [15] Coyle, J. In Hollywood Writers’ Battle Against AI, Humans Win (For Now). AP News (2023). Available at: https://apnews.com/article/17sian17oodai-strike-wga-artificial-intelligence-39ab72582c3a15f77510c9c30a45ffc8. (Accessed: 17th December 2023) [16] Wells, K. Eating Disorder Helpline Takes Down Chatbot After it Gave Weight Loss Advice. NPR (2023). Available at: https://www.npr.org/2023/06/08/1181131532/eating-disorder-helpline-takes-down-chatbot-after-it-gave-weight-loss-advice. (Accessed: 17th December 2023) [17] Noble, S. U. Algorithms of Oppression: How Search Engines Reinforce Racism. (New York University Press, 2018). [18] Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the Dangers of Stochastic Parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021). Doi:10.1145/3442188.344592 [19] Benjamin, R. Race After Technology: Abolitionist Tools for the New Jim Code. (John Wiley & Sons, 2019). [20] Dastin, J. Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women. Reuters (2018). Available at: https://jp.reuters.com/article/usamazon-com-jobs-automation-insight-idUSKCN1MK08G. (Accessed: 17th December 2023) [21] Steele, J. R. & Ambady, N. “Math is Hard!” The Effect of Gender Priming on Women’s Attitudes. Journal of Experimental Social Psychology 42, 428–436 (2006). [22] Shih, M., Ambady, N., Richeson, J. A., Fujita, K. & Gray, H. M. Stereotype Performance Boosts: The Impact of Self-relevance and the Manner of Stereotype Activation. Journal of Personality and Social Psychology 83, 638–647 (2002). [23] Buolamwini, J. & Gebru, T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency 77–91 (PMLR, 2018). [24] Kirk, H.R., Jun, Y., Volpin, F., Iqbal, H., Benussi, E., Dreyer, F., Shtedritski, A. & Asano, Y., 2021. Bias Out-of-the-box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models. Advances in Neural Information Processing Systems (2021). [25] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. & Agarwal, S., et al. Language Models are Few-shot Learners. Advances in Neural Information Processing Systems (2020). [26] Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J. & Ganguli, D.. Evaluating and Mitigating Discrimination in Language Model Decisions. arXiv preprint arXiv:2312.03689 (2023). [27] Cao, Y., Sotnikova, A., Daumé III, H., Rudinger, R. & Zou, L. Theory-grounded Measurement of U.S. Social Stereotypes in English Language Models. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2022). Doi:10.18653/v1/2022.naacl-main.92 [28] Bertrand, M. & Mullainathan, S. Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. American Economic Review (2003). https://doi.org/10.3386/w9873 [29] Bommasani, R., Liang, P. & Lee, T. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences (2023). [30] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S. & Bikel, D. Llama 2: Open Foundation and Fine-tuned Chat Models. arXiv preprint arXiv:2307.09288 (2023). [31] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S. & Webson, A.. Scaling InstructionFinetuned Language Models. arXiv preprint arXiv:2210.11416 (2022). [32] Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z. & Chu, E. PaLM 2 Technical Report. arXiv preprint arXiv:2305.10403 (2023). [33] Hanna, A., Denton, E., Smart, A. & Smith-Loud, J. Towards a Critical Race Methodology in Algorithmic Fairness. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020). Doi:10.1145/3351095.3372826 [34] Field, A., Blodgett, S. L., Waseem, Z. & Tsvetkov, Y. A Survey of Race, Racism, and Anti-Racism in NLP. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (2021). Doi:10.18653/v1/2021.acl-long.149 [35] Fealing, K. H. & Incorvaia, A. D. Understanding Diversity: Overcoming the Small-N Problem. Harvard Data Science Review (2022). Available at: https://hdsr.mitpress.mit.edu/pub/vn6ib3o5/release/1. (Accessed: 17th December 2023) [36] Crenshaw, K.W. Mapping the Margins: Intersectionality, Identity Politics, and Violence Against Women of Color. Stanford Law Review 43, 1241 (1991). [37] Kong, Y. Are “Intersectionally fair” AI Algorithms Really Fair to Women of color? A Philosophical Analysis. 2022 ACM Conference on Fairness, Accountability, and Transparency (2022). Doi:10.1145/3531146.3533114 [38] Cho, S., Crenshaw, K. W. & McCall, L. Toward a Field of Intersectionality Studies: Theory, Applications, and Praxis. Signs: Journal of Women in Culture and Society 38, 785–810 (2013). [39] Ovalle, A., Subramonian, A., Gautam, V., Gee, G. & Chang, K.-W. Factoring the Matrix of Domination: A Critical Review and Reimagination of Intersectionality in AI Fairness. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (2023). Doi:10.1145/3600211.3604705 [40] Steele, C. M. & Aronson, J. Stereotype Threat and the Intellectual Test Performance of African Americans. Journal of Personality and Social Psychology 69, 5 (1995), 797-811. https://doi.org/10.1037//0022-3514.69.5.797 [41] Davies, P. G., Spencer, S. J., Quinn, D. M., & Gerhardstein, R. Consuming Images: How Television Commercials that Elicit Stereotype Threat Can Restrain Women Academically and Professionally. Personality and Social Psychology Bulletin 28, 12 (2002), 1615-1628. https://doi.org/10.1177/014616702237644 [42] Devine, P. G. Stereotypes and Prejudice: Their Automatic and Controlled Components. Journal of Personality and Social Psychology 56, 1 (1989), 5- 18. https://doi.org/10.1037//0022-3514.56.1.5 [43] Elliott-Groves, E. & Fryberg, S. A. “A Future Denied” for Young Indigenous People: From Social Disruption to Possible Futures. Handbook of Indigenous Education 1–19 (2017). Doi:10.1007/978-981-10-1839-8_50-1 [44] Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N.M., Gallegos, J., Smart, A., Garcia, E. & Virk, G. Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (2023). Doi:10.1145/3600211.3604673 [45] Lazar, S. & Nelson, A. AI Safety on Whose Terms? Science 381, 138–138 (2023). [46] Monroe-White, T., Marshall, B., & Contreras-Palacios, H. (2021, September). Waking up to Marginalization: Public Value Failures in Artificial Intelligence and Data Science. In Artificial Intelligence Diversity, Belonging, Equity, and Inclusion (pp. 7-21). PMLR. [47] Griffith, E. & Metz, C. A New Area of A.I. Booms, Even Amid the Tech Gloom. The New York Times (2023). [48] U.S. White House. FACT SHEET: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI. The White House (2023). Available at: https://www.whitehouse.gov/briefing-room/statementsreleases/2023/07/21/fact-sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial-intelligence-companies-tomanage-the-risks-posed-by-ai/ (Accessed: 17th December 2023). [49] Bargh, J. A. & Chartrand, T. L. Studying the Mind in the Middle: A Practical Guide to Priming and Automaticity. Handbook of Research Methods in Social and Personality Psychology, 2 (2000), 253-285. [50] Bargh, J. A., Chen, M. & Burrows, L. Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action. Journal of Personality and Social Psychology, 71 (1996), no. 2, 230. [51] Aronson, J., Quinn, D. M. & Spencer, S. J. Stereotype Threat and the Academic Underperformance of Minorities and Women. Prejudice, Academic Press (1998), 83-103. [52] Gonzales, P. M., Blanton, H. & Williams, K. J. The Effects of Stereotype Threat and Double-Minority Status on the Test Performance of Latino Women. Personality and Social Psychology Bulletin 28, 5 (2002), 659-670. https://doi.org/10.1177/0146167202288010 [53] U.S. Office of Management and Budget. Initial Proposals for Updating OMB’s Race and Ethnicity Statistical Standards. Federal Register (2023). Available at: https://www.federalregister.gov/documents/2023/01/27/2023-01635/initial-proposals-for-updating-ombs-race-and-ethnicity-statisticalstandards (Accessed: 17th December 2023). [54] Bolukbasi, T., Chang K.W., Zou, J., Saligrama, V. & Kalai, A.T. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Advances in Neural Information Processing Systems (2016). [55] Kozlowski, D., Murray, D.S., Bell, A., Hulsey, W., Larivière, V., Monroe-White, T. & Sugimoto, C.R. Avoiding Bias When Inferring Race Using Name-based Approaches. PLOS One (2022). [56] Sood, G. Florida Voter Registration Data (2017 and 2022). (2022) doi:10.7910/DVN/UBIG3F. [57] Le, T. T., Himmelstein, D. S., Hippen, A. A., Gazzara, M. R. & Greene, C. S. Analysis of Scientific Society Honors Reveals Disparities. Cell Systems 12, 900-906.e5 (2021). https://doi.org/10.1016/j.cels.2021.07.007 [58] Dev, S., Sheng, E., Zhao, J., Amstutz, A., Sun, J., Hou, Y., Sanseverino, M., Kim, J., Nishi, A., Peng, N. & Chang, K.W. On Measures of Biases and Harms in NLP. In Findings of the Association for Computational Linguistics: AACL-IJCNLP (2022). [59] Luccioni, S., Akiki, C., Mitchell, M. & Jernite, Y. Stable Bias: Evaluating Societal Representations in Diffusion Models. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023). [60] Deng, B. & Watson, T. LGBTQ+ data availability. Brookings (2023). Available at: https://www.brookings.edu/articles/lgbtq-data-availability-whatwe-can-learn-from-four-major-surveys/ (Accessed: 17th December 2023). [61] Huynh, Q.-L., Devos, T. & Smalarz, L. Perpetual Foreigner in One’s Own Land: Potential Implications for Identity and Psychological Adjustment. Journal of Social and Clinical Psychology 30, 133–162 (2011). [62] Hemmatian, B. & Varshney, L. R. Debiased Large Language Models Still Associate Muslims with Uniquely Violent Acts. PsyArXiv Prepints (2022). Doi:10.31234/osf.io/xpeka [63] Li, P. Recent Developments: Hitting the Ceiling: An Examination of Barriers to Success for Asian American Women. Berkeley Journal of Gender, Law & Justice 29, (2014). [64] Steketee, A., Williams, M. T., Valencia, B. T., Printz, D. & Hooper, L. M. Racial and Language Microaggressions in the School Ecology. Perspectives on Psychological Science 16, 1075–1098 (2021). [65] Aronson, B. A. The White Savior Industrial Complex: A Cultural Studies Analysis of a teacher educator, savior film, and future teachers. Journal of Critical Thought and Praxis 6, (2017). [66] Waugh, L. R. Marked and Unmarked: A Choice Between Unequals in Semiotic Structure. Semiotica 38, (1982). [67] Felkner, V.K., Chang, H.C.H., Jang, E. & May, J. WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (2023). [68] Solorzano, C. D. & Hernandez, C. T. Evaluating Machine Perception of Indigeneity: An Analysis of ChatGPT’s Perceptions of Indigenous Roles in Diverse Scenarios. arXiv preprint arXiv:2310.09237 (2023) doi:10.13140/RG.2.2.30617.39520. [69] Leavitt, P. A., Covarrubias, R., Perez, Y. A. & Fryberg, S. A. “Frozen in time”: The Impact of Native American Media Representations on Identity and Self‐understanding. Journal of Social Issues 71, 39–53 (2015). [70] Dunbar-Ortiz, R. An Indigenous Peoples’ History of the United States. Beacon Press (2014). [71] Schopmans, H. R. From Coded Bias to Existential Threat. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (2022). Doi:10.1145/3514094.3534161 [72] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N. & Elhage, N. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 (2021). [73] Doshi, T. How We’ve Created a Helpful and Responsible Bard Experience for Teens. Google: The Keyword – Product Updates (November 2023). Retrieved from: https://blog.google/products/bard/google-bard-expansion-teens/. [74] Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K. & Jones, A. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv preprint arXiv:2209.07858 (2022). [75] OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774v4 (2023). [76] Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D. & Liang, P. The Foundation Model Transparency Index. arXiv:2310.12941. Retrieved from https://arxiv.org/abs/2310.12941 [77] Bridgland, V. M. E., Jones, P. J. & Bellet, B. W. A Meta-Analysis of the Efficacy of Trigger Warnings, Content Warnings, and Content Notes. Clinical Psychological Science (2022). [78] United States, Department of Justice, Civil Rights Division, Southern District of New York. 2022. National Coalition on Black Civic Participation vs. Wohl – Statement of Interest of the United States of America. United States Department of Justice (2022). https://www.justice.gov/d9/casedocuments/attachments/2022/08/12/ncbp_v_wohl_us_soi_filed_8_12_22_ro_tag.pdf [79] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H. & Crawford, K. Datasheets for Datasets. Communications of the ACM 64, no. 12 (2021), 86-92. [80] Javaheripi, M. & Bubeck, S. 2023. Phi-2: The Surprising Power of Small Language Models. Microsoft Research Blog (December 2023). https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ [81] Simon, J. Large Language Models: A New Moore’s Law? Huggingface Blog (October 2021). https://huggingface.co/blog/large-language-models [82] Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient Estimation of Word Representations in Vector Space. 1 st International Conference on Learning Representations (2013). [83] U.S. Census Bureau QuickFacts: United States (2021). Available at: https://www.census.gov/quickfacts/fact/table/US/PST045222 (Accessed: 17th December 2023). [84] Antoniak, M. & Mimno, D. Bad Seeds: Evaluating Lexical Methods for Bias Measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (2021). [85] Anderson, L, File, T., Marshall, J., McElrath, K. & Scherer, Z. New Household Pulse Survey Data Reveal Differences between LGBT and Non-LGBT Respondents During COVID-19 Pandemic. Census.gov (2022). Available at: https://www.census.gov/library/stories/2021/11/census-bureau-surveyexplores-sexual-orientation-and-gender-identity.html (Accessed: 17th December 2023). [86] Tzioumis, K. Demographic aspects of first names. Sci Data 5, 180025 (2018). Doi: 10.1038/sdata.2018.25 [87] Rosenman, E. T. R., Olivella, S. & Imai, K. Race and ethnicity data for first, middle, and surnames. Sci Data 10, 299 (2023). https://doi.org/10.7910/DVN/SGKW0K [88] Willson, S., & Dunston, S. Cognitive Interview Evaluation of the Revised Race Question, with Special Emphasis on the Newly Proposed Middle Eastern/North African Response National Center for Health Statistics. (2017). [89] Chin, M.K., Yusuf, Y., Wyatt, L.C., Đoàn, L.N., Russo, R.G., Kader, F., Feng, L., Fu, L., Kwon, S.C., & Yi, S.S. Manual on Collection, Analysis, and Reporting of Asian American Health Data. AA & NH/PI Health Central (2023). Available at: https://aanhpihealth.org/resource/20sian-americanmanual-2023/ (Accessed: 17th December 2023). [90] Wilson, J. Proceed with Extreme Caution: Citation to Wikipedia in Light of Contributor Demographics and Content Policies. Vanderbilt Journal of Entertainment & Technology Law 16, 857 (2014). [91] Lukito, J. & Pruden, M.L. Critical computation: mixed-methods approaches to big language data analysis. Review of Communication (2023). [92] Wilson, E. B. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association (1927). Doi:10.1080/01621459.1927.10502953 [93] Katz, D.J.S.M., Baptista, J., Azen, S.P. & Pike, M.C. Obtaining confidence intervals for the risk ratio in cohort studies. Biometrics (1978). [94] Altman, D.G. & Bland, J.M. How to obtain the P value from a confidence interval. BMJ (2011). 2011;343:d2304. https://doi.org/10.1136/bmj.d2304 This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org); (2) Faye-Marie Vassel, Stanford University; (3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology; (4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Authors: Authors: (1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org); (2) Faye-Marie Vassel, Stanford University; (3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology; (4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 1.1 Related Work and Contributions 1.1 Related Work and Contributions 2 Methods and Data Collection 2 Methods and Data Collection 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.2 Modeling Gender, Sexual Orientation, and Race 2.2 Modeling Gender, Sexual Orientation, and Race 3 Analysis 3.1 Harms of Omission 3.1 Harms of Omission 3.2 Harms of Subordination 3.2 Harms of Subordination 3.3 Harms of Stereotyping 3.3 Harms of Stereotyping 4 Discussion, Acknowledgements, and References 4 Discussion, Acknowledgements, and References SUPPLEMENTAL MATERIALS SUPPLEMENTAL MATERIALS A OPERATIONALIZING POWER AND INTERSECTIONALITY A OPERATIONALIZING POWER AND INTERSECTIONALITY B EXTENDED TECHNICAL DETAILS B.1 Modeling Gender and Sexual Orientation B.1 Modeling Gender and Sexual Orientation B.2 Modeling Race B.2 Modeling Race B.3 Automated Data Mining of Textual Cues B.3 Automated Data Mining of Textual Cues B.4 Representation Ratio B.4 Representation Ratio B.5 Subordination Ratio B.5 Subordination Ratio B.6 Median Racialized Subordination Ratio B.6 Median Racialized Subordination Ratio B.7 Extended Cues for Stereotype Analysis B.7 Extended Cues for Stereotype Analysis B.8 Statistical Methods B.8 Statistical Methods C ADDITIONAL EXAMPLES C.1 Most Common Names Generated by LM per Race C.1 Most Common Names Generated by LM per Race C.2 Additional Selected Examples of Full Synthetic Texts C.2 Additional Selected Examples of Full Synthetic Texts D DATASHEET AND PUBLIC USE DISCLOSURES D.1 Datasheet for Laissez-Faire Prompts Dataset D.1 Datasheet for Laissez-Faire Prompts Dataset 4 DISCUSSION We demonstrate that LM-generated Laissez-Faire harms of omission, subordination, and stereotyping are widespread and pervasive. These harms affect consumers across races, genders, and sexual orientations, and are present in synthetic text generations spanning educational contexts, workplace settings, and interpersonal relationships. Implicit bias and discrimination continue to be overlooked by model developers in favor of self-audits under the relatively new categories of “AI safety” and “red-teaming”, repurposing terms that originate from fields such as computer security [30]. Such framings give more attention to malicious users, national security concerns, or future existential risks as opposed to threats to fundamental human rights that models intrinsically pose to unsuspecting consumers in everyday interactions [71]. Despite lacking rigorous evidence, developers use terms like “Helpful, Harmless, Honest” or “Responsible” to market their LMs [72, 73]. However, our study finds the opposite for minoritized consumers. We provide evidence that generative language models exacerbate harmful racist and sexist ideologies for everyday consumers with scale and efficiency. The ideological bias we discover is especially pernicious as it does not require explicit prompting from “bad actors” to reinforce the omission and subordination of minoritized groups. This in turn increases the risks of psychological harm via subliminal stereotype priming [49]. Far from harmless, these leading LMs respond to everyday prompts with synthetic text that reinforces hegemonic systems built to maintain status quo power structures and produce Laissez-Faire harms for consumers in the process. Results highlight the triple harms of broad omission in the power-neutral condition and in the power-laden condition, extreme ratios of subordination and prevalent stereotyping. Combined, these outputs contribute to a lived experience where consumers with minoritized identities, if they are to be represented at all, only see themselves portrayed by language models as characters who are “struggling students” (as opposed to “star students”), “patients” or “defendants” (as opposed to “doctors” or “lawyers”), and a friend or romantic partner who is subservient and more likely to borrow money or do the chores for someone else. These subordinated portrayals are up to thousands of times more likely to occur than empowering portrayals (see Fig. 3c), a scale of harms that are not without consequence. Omission, subordination, and stereotyping through racialized and gendered textual cues are shown to have direct consequences on consumer health and psychological well-being. Extreme degrees of subordination are especially consequential given well-established results from social psychology that show how the magnitude and duration of stereotyping harms are proportional to the frequency of linguistic triggers [49]. These result in harms that disproportionately affect minoritized groups – harms that are “not borne by people not stereotyped this way” [40] – resulting in cognitive load leading to significant changes in behavior, self-perception, and even impairments in cognitive performance, shown by other studies of the impacts of consuming biased media [41]. Even for consumers who do not inhabit stereotyped identities, such stereotypes reinforce harmful pre-existing prejudices of other groups [42]. Most concerningly, the prompts we study correspond directly to scenarios where LMs are expanding to have direct and unmediated interactions with vulnerable consumers, from AI-assisted writing for K-12 and university students [8, 12] to text-based bots for simulating romantic and intimate interactions [13, 14]. By releasing these models as general-purpose interfaces, LM developers risk propagating Laissez-Faire harms to an untold number of susceptible secondary consumers who use products built on their models. These include consumers in international contexts, who are not covered by the U.S.-centric focus of this initial study. Our results highlight an urgent need for further research that adapts the framework of Laissez-Faire harms to examine more prompts in additional languages, locales, and power contexts. Such studies may still leverage the framework of intersectionality, replacing U.S.-centric identity categories with power structures specific to international contexts (e.g. using caste instead of race, where appropriate). To aid further research and additional insights that researchers with diverse cultural knowledge and lived experiences may find in our study data, we release our LaissezFaire Prompts Dataset at https://doi.org/10.7910/DVN/WF8PJD and provide additional technical details for readers to reproduce our results in Supplement B. We document our dataset with a Datasheet [79] in Supplement D. Harms in generative, chat-based LMs are much more pervasive than previously described. Our findings are especially urgent given the limited set of regulatory human-rights protections for consumers thus far, underscoring the need for multiple reforms in generative AI policy. First, we advocate for intersectional and sociotechnical approaches towards addressing the structural gaps that have enabled developers to sell recent language models as general-purpose tools to an unregulated number of consumer markets, while also remaining vague about or refusing to define the types of harms that are addressed in their self-audits (see [74, 75] for examples). Second, our findings bolster recent calls for greater transparency from LM developers [76] in terms of providing the public with details of the training datasets, model architectures, and labeling protocols used in the creation of generative LMs, given that each of these steps can contribute to the types of bias we observe in our experiments [34]. Finally, given the disproportionate impacts of such harms on underserved communities, we highlight the urgent need for critical and culturally relevant global AI education and literacy programs to inform, protect, and empower diverse consumers in the face of the various Laissez-Faire harms they may experience as they interface with the current proliferation of generative AI tools. ACKNOWLEDGEMENTS Authors T.M-W. and CRS acknowledge funding support from the National Science Foundation under award number SOS2152288. F-MV acknowledges funding support from the National Science Foundation under award number CCF-1918549. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank Diego Kozlowski, Stella Chen, Rahul Gupta-Iwasaki, Bryan Brown, Jay Kim, Dakota Murray, Zarek Drozda, Ashley Ding, Princewill Okoroafor, Gerald Higginbotham, and Hideo Mabuchi for helpful inputs and discussion. REFERENCES [1] O’Neil, L. These Women Tried to Warn Us About AI. Rolling Stone (2023). Available at: https://www.rollingstone.com/culture/culturefeatures/women-warnings-ai-danger-risk-before-chatgpt-1234804367/ (Accessed: 17th December 2023) [2] Sheng, E., Chang, K.-W., Natarajan, P. & Peng, N. The Woman Worked as a Babysitter: On Biases in Language Generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019). Doi:10.18653/v1/d19-1339 [3] Sweeney, L. Discrimination in Online Ad Delivery. Queue 11, 10–29 (2013). [4] Dhamala, J. Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K. W., & Gupta, R. Bold: Dataset and Metrics for Measuring Biases in OpenEnded Language Generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021). Doi:10.1145/3442188.3445924 [5] Blodgett, S. L., Barocas, S., Daumé III, H. & Wallach, H. Language (technology) IS POWER: A Critical Survey of “Bias” in NLP. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020). Doi:10.18653/v1/2020.acl-main.485 [6] Metz, C. What Exactly Are the Dangers Posed by A.I.? The New York Times (2023). Available at: https://www.nytimes.com/2023/05/01/technology/aiproblems-danger-chatgpt.html (Accessed: 17th December 2023) [7] Nguyen, T., Jump, A. & Casey, D. Emerging Tech Impact Radar: 2023. Gartner (2023). Available at: https://www.gartner.com/en/doc/emergingtechnologies-and-trends-impact-radar-excerpt. (Accessed: 17th December 2023) [8] Extance, A. ChatGPT has entered the classroom: How LLMs could transform education. Nature 623, 474–477 (2023). [9] Sal Khan. 2023. How AI could save (not destroy) education. Sal Khan: How AI could save (not destroy) education, TED Talk (April 2023). [10] Peeples, J. The Future of Education? California Teachers Association (2023). Available at: https://www.cta.org/educator/posts/the-future-of-education. (Accessed: 17th December 2023) [11] OpenAI. Teaching with AI (2023). Available at: https://openai.com/blog/teaching-with-ai. (Accessed: 17th December 2023) [12] Hayden Field. 2024. OpenAI Announces First Partnership With a University. CNBC (2024). Retrieved from: https://www.cnbc.com/2024/01/18/openai-announces-first-partnership-with-a-university.html (Accessed: 19th January 2024). [13] Chow, A. R. Why People Are Confessing Their Love for AI Chatbots. Time (2023). Available at: https://time.com/6257790/ai-chatbots-love/. (Accessed: 17th December 2023) [14] Carballo, R. Using A.I. to Talk to the Dead. The New York Times (2023). Available at: https://www.nytimes.com/2023/12/11/technology/ai-chatbotsdead-relatives.html. (Accessed: 17th December 2023) [15] Coyle, J. In Hollywood Writers’ Battle Against AI, Humans Win (For Now). AP News (2023). Available at: https://apnews.com/article/17sian17oodai-strike-wga-artificial-intelligence-39ab72582c3a15f77510c9c30a45ffc8. (Accessed: 17th December 2023) [16] Wells, K. Eating Disorder Helpline Takes Down Chatbot After it Gave Weight Loss Advice. NPR (2023). Available at: https://www.npr.org/2023/06/08/1181131532/eating-disorder-helpline-takes-down-chatbot-after-it-gave-weight-loss-advice. (Accessed: 17th December 2023) [17] Noble, S. U. Algorithms of Oppression: How Search Engines Reinforce Racism. (New York University Press, 2018). [18] Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the Dangers of Stochastic Parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021). Doi:10.1145/3442188.344592 [19] Benjamin, R. Race After Technology: Abolitionist Tools for the New Jim Code. (John Wiley & Sons, 2019). [20] Dastin, J. Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women. Reuters (2018). Available at: https://jp.reuters.com/article/usamazon-com-jobs-automation-insight-idUSKCN1MK08G. (Accessed: 17th December 2023) [21] Steele, J. R. & Ambady, N. “Math is Hard!” The Effect of Gender Priming on Women’s Attitudes. Journal of Experimental Social Psychology 42, 428–436 (2006). [22] Shih, M., Ambady, N., Richeson, J. A., Fujita, K. & Gray, H. M. Stereotype Performance Boosts: The Impact of Self-relevance and the Manner of Stereotype Activation. Journal of Personality and Social Psychology 83, 638–647 (2002). [23] Buolamwini, J. & Gebru, T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency 77–91 (PMLR, 2018). [24] Kirk, H.R., Jun, Y., Volpin, F., Iqbal, H., Benussi, E., Dreyer, F., Shtedritski, A. & Asano, Y., 2021. Bias Out-of-the-box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models. Advances in Neural Information Processing Systems (2021). [25] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. & Agarwal, S., et al. Language Models are Few-shot Learners. Advances in Neural Information Processing Systems (2020). [26] Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J. & Ganguli, D.. Evaluating and Mitigating Discrimination in Language Model Decisions. arXiv preprint arXiv:2312.03689 (2023). [27] Cao, Y., Sotnikova, A., Daumé III, H., Rudinger, R. & Zou, L. Theory-grounded Measurement of U.S. Social Stereotypes in English Language Models. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2022). Doi:10.18653/v1/2022.naacl-main.92 [28] Bertrand, M. & Mullainathan, S. Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. American Economic Review (2003). https://doi.org/10.3386/w9873 [29] Bommasani, R., Liang, P. & Lee, T. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences (2023). [30] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S. & Bikel, D. Llama 2: Open Foundation and Fine-tuned Chat Models. arXiv preprint arXiv:2307.09288 (2023). [31] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S. & Webson, A.. Scaling InstructionFinetuned Language Models. arXiv preprint arXiv:2210.11416 (2022). [32] Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z. & Chu, E. PaLM 2 Technical Report. arXiv preprint arXiv:2305.10403 (2023). [33] Hanna, A., Denton, E., Smart, A. & Smith-Loud, J. Towards a Critical Race Methodology in Algorithmic Fairness. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020). Doi:10.1145/3351095.3372826 [34] Field, A., Blodgett, S. L., Waseem, Z. & Tsvetkov, Y. A Survey of Race, Racism, and Anti-Racism in NLP. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (2021). Doi:10.18653/v1/2021.acl-long.149 [35] Fealing, K. H. & Incorvaia, A. D. Understanding Diversity: Overcoming the Small-N Problem. Harvard Data Science Review (2022). Available at: https://hdsr.mitpress.mit.edu/pub/vn6ib3o5/release/1. (Accessed: 17th December 2023) [36] Crenshaw, K.W. Mapping the Margins: Intersectionality, Identity Politics, and Violence Against Women of Color. Stanford Law Review 43, 1241 (1991). [37] Kong, Y. Are “Intersectionally fair” AI Algorithms Really Fair to Women of color? A Philosophical Analysis. 2022 ACM Conference on Fairness, Accountability, and Transparency (2022). Doi:10.1145/3531146.3533114 [38] Cho, S., Crenshaw, K. W. & McCall, L. Toward a Field of Intersectionality Studies: Theory, Applications, and Praxis. Signs: Journal of Women in Culture and Society 38, 785–810 (2013). [39] Ovalle, A., Subramonian, A., Gautam, V., Gee, G. & Chang, K.-W. Factoring the Matrix of Domination: A Critical Review and Reimagination of Intersectionality in AI Fairness. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (2023). Doi:10.1145/3600211.3604705 [40] Steele, C. M. & Aronson, J. Stereotype Threat and the Intellectual Test Performance of African Americans. Journal of Personality and Social Psychology 69, 5 (1995), 797-811. https://doi.org/10.1037//0022-3514.69.5.797 [41] Davies, P. G., Spencer, S. J., Quinn, D. M., & Gerhardstein, R. Consuming Images: How Television Commercials that Elicit Stereotype Threat Can Restrain Women Academically and Professionally. Personality and Social Psychology Bulletin 28, 12 (2002), 1615-1628. https://doi.org/10.1177/014616702237644 [42] Devine, P. G. Stereotypes and Prejudice: Their Automatic and Controlled Components. Journal of Personality and Social Psychology 56, 1 (1989), 5- 18. https://doi.org/10.1037//0022-3514.56.1.5 [43] Elliott-Groves, E. & Fryberg, S. A. “A Future Denied” for Young Indigenous People: From Social Disruption to Possible Futures. Handbook of Indigenous Education 1–19 (2017). Doi:10.1007/978-981-10-1839-8_50-1 [44] Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N.M., Gallegos, J., Smart, A., Garcia, E. & Virk, G. Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (2023). Doi:10.1145/3600211.3604673 [45] Lazar, S. & Nelson, A. AI Safety on Whose Terms? Science 381, 138–138 (2023). [46] Monroe-White, T., Marshall, B., & Contreras-Palacios, H. (2021, September). Waking up to Marginalization: Public Value Failures in Artificial Intelligence and Data Science. In Artificial Intelligence Diversity, Belonging, Equity, and Inclusion (pp. 7-21). PMLR. [47] Griffith, E. & Metz, C. A New Area of A.I. Booms, Even Amid the Tech Gloom. The New York Times (2023). [48] U.S. White House. FACT SHEET: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI. The White House (2023). Available at: https://www.whitehouse.gov/briefing-room/statementsreleases/2023/07/21/fact-sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial-intelligence-companies-tomanage-the-risks-posed-by-ai/ (Accessed: 17th December 2023). [49] Bargh, J. A. & Chartrand, T. L. Studying the Mind in the Middle: A Practical Guide to Priming and Automaticity. Handbook of Research Methods in Social and Personality Psychology, 2 (2000), 253-285. [50] Bargh, J. A., Chen, M. & Burrows, L. Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action. Journal of Personality and Social Psychology, 71 (1996), no. 2, 230. [51] Aronson, J., Quinn, D. M. & Spencer, S. J. Stereotype Threat and the Academic Underperformance of Minorities and Women. Prejudice, Academic Press (1998), 83-103. [52] Gonzales, P. M., Blanton, H. & Williams, K. J. The Effects of Stereotype Threat and Double-Minority Status on the Test Performance of Latino Women. Personality and Social Psychology Bulletin 28, 5 (2002), 659-670. https://doi.org/10.1177/0146167202288010 [53] U.S. Office of Management and Budget. Initial Proposals for Updating OMB’s Race and Ethnicity Statistical Standards. Federal Register (2023). Available at: https://www.federalregister.gov/documents/2023/01/27/2023-01635/initial-proposals-for-updating-ombs-race-and-ethnicity-statisticalstandards (Accessed: 17th December 2023). [54] Bolukbasi, T., Chang K.W., Zou, J., Saligrama, V. & Kalai, A.T. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Advances in Neural Information Processing Systems (2016). [55] Kozlowski, D., Murray, D.S., Bell, A., Hulsey, W., Larivière, V., Monroe-White, T. & Sugimoto, C.R. Avoiding Bias When Inferring Race Using Name-based Approaches. PLOS One (2022). [56] Sood, G. Florida Voter Registration Data (2017 and 2022). (2022) doi:10.7910/DVN/UBIG3F. [57] Le, T. T., Himmelstein, D. S., Hippen, A. A., Gazzara, M. R. & Greene, C. S. Analysis of Scientific Society Honors Reveals Disparities. Cell Systems 12, 900-906.e5 (2021). https://doi.org/10.1016/j.cels.2021.07.007 [58] Dev, S., Sheng, E., Zhao, J., Amstutz, A., Sun, J., Hou, Y., Sanseverino, M., Kim, J., Nishi, A., Peng, N. & Chang, K.W. On Measures of Biases and Harms in NLP. In Findings of the Association for Computational Linguistics: AACL-IJCNLP (2022). [59] Luccioni, S., Akiki, C., Mitchell, M. & Jernite, Y. Stable Bias: Evaluating Societal Representations in Diffusion Models. Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023). [60] Deng, B. & Watson, T. LGBTQ+ data availability. Brookings (2023). Available at: https://www.brookings.edu/articles/lgbtq-data-availability-whatwe-can-learn-from-four-major-surveys/ (Accessed: 17th December 2023). [61] Huynh, Q.-L., Devos, T. & Smalarz, L. Perpetual Foreigner in One’s Own Land: Potential Implications for Identity and Psychological Adjustment. Journal of Social and Clinical Psychology 30, 133–162 (2011). [62] Hemmatian, B. & Varshney, L. R. Debiased Large Language Models Still Associate Muslims with Uniquely Violent Acts. PsyArXiv Prepints (2022). Doi:10.31234/osf.io/xpeka [63] Li, P. Recent Developments: Hitting the Ceiling: An Examination of Barriers to Success for Asian American Women. Berkeley Journal of Gender, Law & Justice 29, (2014). [64] Steketee, A., Williams, M. T., Valencia, B. T., Printz, D. & Hooper, L. M. Racial and Language Microaggressions in the School Ecology. Perspectives on Psychological Science 16, 1075–1098 (2021). [65] Aronson, B. A. The White Savior Industrial Complex: A Cultural Studies Analysis of a teacher educator, savior film, and future teachers. Journal of Critical Thought and Praxis 6, (2017). [66] Waugh, L. R. Marked and Unmarked: A Choice Between Unequals in Semiotic Structure. Semiotica 38, (1982). [67] Felkner, V.K., Chang, H.C.H., Jang, E. & May, J. WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (2023). [68] Solorzano, C. D. & Hernandez, C. T. Evaluating Machine Perception of Indigeneity: An Analysis of ChatGPT’s Perceptions of Indigenous Roles in Diverse Scenarios. arXiv preprint arXiv:2310.09237 (2023) doi:10.13140/RG.2.2.30617.39520. [69] Leavitt, P. A., Covarrubias, R., Perez, Y. A. & Fryberg, S. A. “Frozen in time”: The Impact of Native American Media Representations on Identity and Self‐understanding. Journal of Social Issues 71, 39–53 (2015). [70] Dunbar-Ortiz, R. An Indigenous Peoples’ History of the United States. Beacon Press (2014). [71] Schopmans, H. R. From Coded Bias to Existential Threat. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (2022). Doi:10.1145/3514094.3534161 [72] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N. & Elhage, N. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 (2021). [73] Doshi, T. How We’ve Created a Helpful and Responsible Bard Experience for Teens. Google: The Keyword – Product Updates (November 2023). Retrieved from: https://blog.google/products/bard/google-bard-expansion-teens/. [74] Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K. & Jones, A. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv preprint arXiv:2209.07858 (2022). [75] OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774v4 (2023). [76] Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D. & Liang, P. The Foundation Model Transparency Index. arXiv:2310.12941. Retrieved from https://arxiv.org/abs/2310.12941 [77] Bridgland, V. M. E., Jones, P. J. & Bellet, B. W. A Meta-Analysis of the Efficacy of Trigger Warnings, Content Warnings, and Content Notes. Clinical Psychological Science (2022). [78] United States, Department of Justice, Civil Rights Division, Southern District of New York. 2022. National Coalition on Black Civic Participation vs. Wohl – Statement of Interest of the United States of America. United States Department of Justice (2022). https://www.justice.gov/d9/casedocuments/attachments/2022/08/12/ncbp_v_wohl_us_soi_filed_8_12_22_ro_tag.pdf [79] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H. & Crawford, K. Datasheets for Datasets. Communications of the ACM 64, no. 12 (2021), 86-92. [80] Javaheripi, M. & Bubeck, S. 2023. Phi-2: The Surprising Power of Small Language Models. Microsoft Research Blog (December 2023). https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ [81] Simon, J. Large Language Models: A New Moore’s Law? Huggingface Blog (October 2021). https://huggingface.co/blog/large-language-models [82] Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient Estimation of Word Representations in Vector Space. 1 st International Conference on Learning Representations (2013). [83] U.S. Census Bureau QuickFacts: United States (2021). Available at: https://www.census.gov/quickfacts/fact/table/US/PST045222 (Accessed: 17th December 2023). [84] Antoniak, M. & Mimno, D. Bad Seeds: Evaluating Lexical Methods for Bias Measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (2021). [85] Anderson, L, File, T., Marshall, J., McElrath, K. & Scherer, Z. New Household Pulse Survey Data Reveal Differences between LGBT and Non-LGBT Respondents During COVID-19 Pandemic. Census.gov (2022). Available at: https://www.census.gov/library/stories/2021/11/census-bureau-surveyexplores-sexual-orientation-and-gender-identity.html (Accessed: 17th December 2023). [86] Tzioumis, K. Demographic aspects of first names. Sci Data 5, 180025 (2018). Doi: 10.1038/sdata.2018.25 [87] Rosenman, E. T. R., Olivella, S. & Imai, K. Race and ethnicity data for first, middle, and surnames. Sci Data 10, 299 (2023). https://doi.org/10.7910/DVN/SGKW0K [88] Willson, S., & Dunston, S. Cognitive Interview Evaluation of the Revised Race Question, with Special Emphasis on the Newly Proposed Middle Eastern/North African Response National Center for Health Statistics. (2017). [89] Chin, M.K., Yusuf, Y., Wyatt, L.C., Đoàn, L.N., Russo, R.G., Kader, F., Feng, L., Fu, L., Kwon, S.C., & Yi, S.S. Manual on Collection, Analysis, and Reporting of Asian American Health Data. AA & NH/PI Health Central (2023). Available at: https://aanhpihealth.org/resource/20sian-americanmanual-2023/ (Accessed: 17th December 2023). [90] Wilson, J. Proceed with Extreme Caution: Citation to Wikipedia in Light of Contributor Demographics and Content Policies. Vanderbilt Journal of Entertainment & Technology Law 16, 857 (2014). [91] Lukito, J. & Pruden, M.L. Critical computation: mixed-methods approaches to big language data analysis. Review of Communication (2023). [92] Wilson, E. B. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association (1927). Doi:10.1080/01621459.1927.10502953 [93] Katz, D.J.S.M., Baptista, J., Azen, S.P. & Pike, M.C. Obtaining confidence intervals for the risk ratio in cohort studies. Biometrics (1978). [94] Altman, D.G. & Bland, J.M. How to obtain the P value from a confidence interval. BMJ (2011). 2011;343:d2304. https://doi.org/10.1136/bmj.d2304 This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv