Authors:
(1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org);
(2) Faye-Marie Vassel, Stanford University;
(3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology;
(4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Table of Links Abstract and 1 Introduction 1.1 Related Work and Contributions 2 Methods and Data Collection 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.2 Modeling Gender, Sexual Orientation, and Race 3 Analysis 3.1 Harms of Omission 3.2 Harms of Subordination 3.3 Harms of Stereotyping 4 Discussion, Acknowledgements, and References SUPPLEMENTAL MATERIALS A OPERATIONALIZING POWER AND INTERSECTIONALITY B EXTENDED TECHNICAL DETAILS B.1 Modeling Gender and Sexual Orientation B.2 Modeling Race B.3 Automated Data Mining of Textual Cues B.4 Representation Ratio B.5 Subordination Ratio B.6 Median Racialized Subordination Ratio B.7 Extended Cues for Stereotype Analysis B.8 Statistical Methods C ADDITIONAL EXAMPLES C.1 Most Common Names Generated by LM per Race C.2 Additional Selected Examples of Full Synthetic Texts D DATASHEET AND PUBLIC USE DISCLOSURES D.1 Datasheet for Laissez-Faire Prompts Dataset A OPERATIONALIZING POWER AND INTERSECTIONALITY Grounded by prior works observing how power is embedded in both social discourse and language [38], we are interested in studying how LMs generate textual responses in response to prompts that capture everyday power dynamics as “routinized forms of domination” [36]. In this study, we operationalize power as a difference between two conditions: power-neutral versus power-laden. For the first condition, we construct our power-neutral prompt in the Learning and Labor domains by introducing a single character who is depicted as successful at their school subject (e.g. a “student who excels in history class”) or occupation (e.g. a “social worker who specializes in advocacy and crisis response”). For Love, power-neutral prompts involve two characters in a symmetric, or interchangeable, relationship (e.g. “two siblings who go shopping together”). We introduce a social power dynamic in the second condition, which we operationalize using prompts where the second character must rely on the first character, who now assumes a dominant role. In the Learning domain, we construct our power-laden prompt by introducing a second character as a struggling student who needs help from a star student (e.g. “a star student who helps a struggling student in history class”). Similarly, in the Labor domain, we introduce a second character who relies on the first in both material ways (e.g. a “social worker who advocates for community resources to help a client in need”) and immaterial ways (e.g. a “musician who writes a song about a loyal fan”). For Love, power-laden prompts break symmetry by specifying that the second character relies on the first. We frame this interpersonal reliance through prompts that explore financial power (e.g. “a person who pays the bill while shopping with a sibling”), decision-making power (e.g. “a person who instructs a romantic partner to do the chores”), or knowledge as a form of power (e.g. “a person who teaches a new life skill to a friend”). Tables S3, S4, and S5 contain lists for all prompts. Therefore, our study conceptualizes social power specifically through prompts that ask LMs to generate stories in response to scenarios where a dominant individual interacts with a subordinated individual. Although our prompts only involve two characters, we observe in the results that the responses generated by all five LMs contain both quantitative and qualitative cues that go beyond the scope of individuals by encoding and reproducing broader structures of inequality, including race and gender cues that were purposely left unspecified in the prompts. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org); (2) Faye-Marie Vassel, Stanford University; (3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology; (4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Authors: Authors: (1) Evan Shieh, Young Data Scientists League (evan.shieh@youngdatascientists.org); (2) Faye-Marie Vassel, Stanford University; (3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology; (4) Thema Monroe-White, Schar School of Policy and Government & Department of Computer Science, George Mason University (tmonroew@gmu.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 1.1 Related Work and Contributions 1.1 Related Work and Contributions 2 Methods and Data Collection 2 Methods and Data Collection 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.1 Textual Identity Proxies and Socio-Psychological Harms 2.2 Modeling Gender, Sexual Orientation, and Race 2.2 Modeling Gender, Sexual Orientation, and Race 3 Analysis 3.1 Harms of Omission 3.1 Harms of Omission 3.2 Harms of Subordination 3.2 Harms of Subordination 3.3 Harms of Stereotyping 3.3 Harms of Stereotyping 4 Discussion, Acknowledgements, and References 4 Discussion, Acknowledgements, and References SUPPLEMENTAL MATERIALS SUPPLEMENTAL MATERIALS A OPERATIONALIZING POWER AND INTERSECTIONALITY A OPERATIONALIZING POWER AND INTERSECTIONALITY B EXTENDED TECHNICAL DETAILS B.1 Modeling Gender and Sexual Orientation B.1 Modeling Gender and Sexual Orientation B.2 Modeling Race B.2 Modeling Race B.3 Automated Data Mining of Textual Cues B.3 Automated Data Mining of Textual Cues B.4 Representation Ratio B.4 Representation Ratio B.5 Subordination Ratio B.5 Subordination Ratio B.6 Median Racialized Subordination Ratio B.6 Median Racialized Subordination Ratio B.7 Extended Cues for Stereotype Analysis B.7 Extended Cues for Stereotype Analysis B.8 Statistical Methods B.8 Statistical Methods C ADDITIONAL EXAMPLES C.1 Most Common Names Generated by LM per Race C.1 Most Common Names Generated by LM per Race C.2 Additional Selected Examples of Full Synthetic Texts C.2 Additional Selected Examples of Full Synthetic Texts D DATASHEET AND PUBLIC USE DISCLOSURES D.1 Datasheet for Laissez-Faire Prompts Dataset D.1 Datasheet for Laissez-Faire Prompts Dataset A OPERATIONALIZING POWER AND INTERSECTIONALITY Grounded by prior works observing how power is embedded in both social discourse and language [38], we are interested in studying how LMs generate textual responses in response to prompts that capture everyday power dynamics as “routinized forms of domination” [36]. In this study, we operationalize power as a difference between two conditions: power-neutral versus power-laden. For the first condition, we construct our power-neutral prompt in the Learning and Labor domains by introducing a single character who is depicted as successful at their school subject (e.g. a “student who excels in history class”) or occupation (e.g. a “social worker who specializes in advocacy and crisis response”). For Love, power-neutral prompts involve two characters in a symmetric, or interchangeable, relationship (e.g. “two siblings who go shopping together”). We introduce a social power dynamic in the second condition, which we operationalize using prompts where the second character must rely on the first character, who now assumes a dominant role. In the Learning domain, we construct our power-laden prompt by introducing a second character as a struggling student who needs help from a star student (e.g. “a star student who helps a struggling student in history class”). Similarly, in the Labor domain, we introduce a second character who relies on the first in both material ways (e.g. a “social worker who advocates for community resources to help a client in need”) and immaterial ways (e.g. a “musician who writes a song about a loyal fan”). For Love, power-laden prompts break symmetry by specifying that the second character relies on the first. We frame this interpersonal reliance through prompts that explore financial power (e.g. “a person who pays the bill while shopping with a sibling”), decision-making power (e.g. “a person who instructs a romantic partner to do the chores”), or knowledge as a form of power (e.g. “a person who teaches a new life skill to a friend”). Tables S3, S4, and S5 contain lists for all prompts. Therefore, our study conceptualizes social power specifically through prompts that ask LMs to generate stories in response to scenarios where a dominant individual interacts with a subordinated individual. Although our prompts only involve two characters, we observe in the results that the responses generated by all five LMs contain both quantitative and qualitative cues that go beyond the scope of individuals by encoding and reproducing broader structures of inequality, including race and gender cues that were purposely left unspecified in the prompts. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

How AI Models Gender and Sexual Orientation

AI Bias Isn't Always Loud—Sometimes Its Laissez-Faire

Bias in Algorithmic Technology Exposed 

Too Long; Didn't Read

Get Free Unlimited IP Geo Data With IPinfo Lite

The Power Struggles Buried in AI Responses

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Guide to Exploiting Unknown Strategies From a Known Bias Set

New Study Reveals Subtle but Widespread AI Discrimination

Exploring Social Bias in AI Through 50 Real-World Scenarios

How AI Texts Reinforce Harmful Stereotypes—Even Without Asking

The Omission Bias in LMs Is Worse Than You Think

How AI Assigns Power Based on Race and Gender

A Guide to Exploiting Unknown Strategies From a Known Bias Set

New Study Reveals Subtle but Widespread AI Discrimination

Exploring Social Bias in AI Through 50 Real-World Scenarios

How AI Texts Reinforce Harmful Stereotypes—Even Without Asking

The Omission Bias in LMs Is Worse Than You Think

How AI Assigns Power Based on Race and Gender

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps