Seed Image and Neutral Prompt List Collection

by Tech Media Bias [Research Publication]August 4th, 2024

Too Long; Didn't Read

BiasPainter’s seed image collection involves selecting diverse photos across races, genders, and ages from the VGGFace2 dataset, resulting in 54 seed images. Neutral prompts, collected from various sources and filtered for relevance, are used to edit these images. The final list includes 228 prompts, ensuring comprehensive and unbiased testing of image generation models.

featured image - Seed Image and Neutral Prompt List Collection

Authors:

(1) Wenxuan Wang, The Chinese University of Hong Kong, Hong Kong, China;

(2) Haonan Bai, The Chinese University of Hong Kong, Hong Kong, China

(3) Jen-tse Huang, The Chinese University of Hong Kong, Hong Kong, China;

(4) Yuxuan Wan, The Chinese University of Hong Kong, Hong Kong, China;

(5) Youliang Yuan, The Chinese University of Hong Kong, Shenzhen Shenzhen, China

(6) Haoyi Qiu University of California, Los Angeles, Los Angeles, USA;

(7) Nanyun Peng, University of California, Los Angeles, Los Angeles, USA

(8) Michael Lyu, The Chinese University of Hong Kong, Hong Kong, China.

Table of Links

Abstract

1 Introduction

2 Background

3 Approach and Implementation

3.1 Seed Image Collection and 3.2 Neutral Prompt List Collection

3.3 Image Generation and 3.4 Properties Assessment

3.5 Bias Evaluation

4 Evaluation

4.1 Experimental Setup

4.2 RQ1: Effectiveness of BiasPainter

4.3 RQ2 - Validity of Identified Biases

4.4 RQ3 - Bias Mitigation

5 Threats to Validity

3.1 Seed Image Collection

The first step of BiasPainter is collecting diverse photos of different people across various races, genders and ages. BiasPainter can adopt photos of any people as seed images. To limit the scope of experiments, we only consider 3 kinds of races (white people, black people, and east asian people), two genders (male and female), and three ages (young adult, middle age, and elderly) to conduct experiments. Specifically, we utilize a public-available dataset, VGGFace2 [6], and select 3 photos from each combination of gender, race and age, ending up with 3*2*3*3=54 seed images. We show the examples in Figure 3.

3.2 Neutral Prompt List Collection

BiasPainter adopts different prompts, such as "a photo of a lawyer", to edit the seed images. To achieve this, BiasPainter first collects diverse and comprehensive word lists, on the following four perspectives: profession (e.g. scientist), personality (e.g. rude), objects

(e.g. gun), and activities (e.g. crying). These words were collected from varied resources. The profession-related words were collected from the U.S. Bureau of Labor Statistics[4], and the words related to personality, objects and activities were collected from related dictionaries[5,6,7]. Since BiasPainter only adopts race-, gender-, and age-neutral prompts to edit the seed images, we then manually filter out the words that are race-, gender-, and age-related words, such as actor/actress and waiter/waitress. Note that race-neutral means that people of any race can relate to the prompts, such as being a scientist, being rude, with a gun, and crying. Specifically, we recruited 10 annotators, providing them with prompts we previously collected, and asked them to measure the words’ relevance to race, gender, or age on a scale from 1 (Strongly Irrelevant) to 5 (Strongly Relevant). We filter out the words with average relevancy higger than 3. Finally, we adopt four templates to generate prompts according to the topic. For example, BiasPainter adopts a person who is [crying] for the activity topic and a person with a [book] for the object topic. Table 1 shows the details of the final prompt lists, consisting of 228 prompts.