Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 1.1 The twincode platform 1.1 The twincode platform 1.2 Pilot Studies 1.2 Pilot Studies 1.3 Other Gender Identities and 1.4 Structure of the Paper 1.3 Other Gender Identities and 1.4 Structure of the Paper 2 Related Work 2 Related Work 3 Original Study (Seville Dec, 2021) and 3.1 Participants 3 Original Study (Seville Dec, 2021) and 3.1 Participants 3.2 Experiment Execution 3.2 Experiment Execution 3.3 Factors (Independent Variables) 3.3 Factors (Independent Variables) 3.4 Response Variables (Dependent Variables) 3.4 Response Variables (Dependent Variables) 3.5 Confounding Variables 3.5 Confounding Variables 3.6 Data Analysis 3.6 Data Analysis 4 First Replication (Berkeley May, 2022) 4 First Replication (Berkeley May, 2022) 4.1 Participants 4.1 Participants 4.2 Experiment Execution 4.2 Experiment Execution 4.3 Data Analysis 4.3 Data Analysis 5 Discussion and Threats to Validity and 5.1 Operationalization of the Cause Construct — Treatment 5 Discussion and Threats to Validity and 5.1 Operationalization of the Cause Construct — Treatment 5.2 Operationalization of the Effect Construct — Metrics 5.2 Operationalization of the Effect Construct — Metrics 5.3 Sampling the Population — Participants 5.3 Sampling the Population — Participants 6 Conclusions and Future Work 6 Conclusions and Future Work 6.1 Replication in Different Cultural Background 6.1 Replication in Different Cultural Background 6.2 Using Chatbots as Partners and AI-based Utterance Coding 6.2 Using Chatbots as Partners and AI-based Utterance Coding Datasets, Compliance with Ethical Standards, Acknowledgements, and References Datasets, Compliance with Ethical Standards, Acknowledgements, and References A. Questionnaire #1 and #2 response items A. Questionnaire #1 and #2 response items B. Evolution of the twincode User Interface B. Evolution of the twincode User Interface C. User Interface of tag-a-chat C. User Interface of tag-a-chat 3.2 Experiment Execution Some weeks before experiment execution, in order to recruit participants, the students enrolled in the three groups of the Requirements Engineering course taught in Spanish were motivated to voluntarily participate in the study as an interesting experience in remote pair programming, but without mentioning neither that the main goal was to study the potential effect of gender bias, nor they were going to be paired with the same classmate during all the study. We also remarked that for the purpose of the study, they must remain anonymous to their partners, so they must neither mention nor ask any personal information, thus not discovering that their partners were always the same person. After providing all that information, including that the participation in the study counted for a 5% bonus on their grades to prevent dropout, the interested students registered in the twincode platform providing some demographic data and accepting the participation conditions. The experiment execution, which is graphically represented in Figure 2 and 3, took place the same day for the three groups of students of the course during their laboratory sessions, as shown in Figure 4 [5]. All registered students logged into the twincode platform, which automatically allocated them into the control and experimental groups balancing the proportion of women in each group as much as possible. Once all the students were allocated to groups, they were randomly allocated into control-experimental pairs by the platform (see Figure 2). After subject allocation, the pairs were presented a programming exercise that they had to solve collaboratively using twincode (labeled as Task#1 in Figure 3). They were given 10 minutes to solve a first exercise and another 10 minutes to solve a second exercise, thus a total time of 20 minutes. After the first 10-minute period, the second exercise was presented independently of whether the first one was finished successfully or not. Both exercises were randomly selected from a pool of exercises of similar complexity. During this programming exercise in pairs, subjects in the control group received no information about the gender of their partners, whereas subjects in the experimental group could see their partners as having a clearly gendered avatar randomly selected by the platform (see Figure 1). At the end of the 20-minute period, they were asked to individually fill in a questionnaire (labeled as Quest.#1 in Figure 3) about the perceived productivity compared to solo programming, the perceived partner’s technical competency compared to their own, and about the partner’s positive and negative aspects. They were given 10 minutes to fill in the questionnaire. After filling the first questionnaire, the students were presented another programming exercise to be solved individually in 10 minutes (labeled as Task#2 in Figure 3). In the case they finished earlier, another exercise of similar complexity was randomly presented. The main purpose of this individual task was to make students forget about their first partners, i.e. their style of writing chat utterances or source code, so they did not recognize them in the second in-pair task. After the individual task, pairs were presented again a new collaborative programming exercise that they must solve in similar conditions to the exercise in Task#1. In this second in-pair exercise, the gendered avatar was swapped with respect to the first exercise for the subjects in the experimental group. For those in the control group, they continued to receive no information on their partners’ genders. Note that pairs were kept the same in order to reduce the variability due to the subjects themselves, which could possibly have had a confounding effect in case of a new pair allocation for Task#3 (see Section 3.5.1 for details). Once Task#3 was finished, students were asked to fill a questionnaire (labeled as Quest.#2 in Figure 3) with the same questions than the one they filled after Task#1 but referred to the second partner, and another questionnaire (labeled as Quest.#3 in Figure 3) comparing the skills of the first and second partners and whether they remembered the gendered avatars of their partners or not. They were given 15 minutes for responding both questionnaires. Finally, they were informed about the actual purpose of the study. At that point, they were allowed to withdraw their data if they wished, although none of them opted for doing so. Authors:
(1) Amador Duran, I3US Institute, Universidad de Sevilla, Sevilla, Spain and SCORE Lab, Universidad de Sevilla, Sevilla, Spain (amador@us.es);
(2) Pablo Fernandez, I3US Institute, Universidad de Sevilla, Sevilla, Spain and SCORE Lab, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es);
(3) Beatriz Bernardez, I3US Institute, Universidad de Sevilla, Sevilla, Spain and SCORE Lab, Universidad de Sevilla, Sevilla, Spain (beat@us.es);
(4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, USA (nweinman@berkeley.edu);
(5) Aslıhan Akalın, Computer Science Division, University of California, Berkeley, Berkeley, USA (asliakalin@berkeley.edu);
(6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, USA (fox@berkeley.edu). Authors: Authors: (1) Amador Duran, I3US Institute, Universidad de Sevilla, Sevilla, Spain and SCORE Lab, Universidad de Sevilla, Sevilla, Spain (amador@us.es); (2) Pablo Fernandez, I3US Institute, Universidad de Sevilla, Sevilla, Spain and SCORE Lab, Universidad de Sevilla, Sevilla, Spain (pablofm@us.es); (3) Beatriz Bernardez, I3US Institute, Universidad de Sevilla, Sevilla, Spain and SCORE Lab, Universidad de Sevilla, Sevilla, Spain (beat@us.es); (4) Nathaniel Weinman, Computer Science Division, University of California, Berkeley, Berkeley, USA (nweinman@berkeley.edu); (5) Aslıhan Akalın, Computer Science Division, University of California, Berkeley, Berkeley, USA (asliakalin@berkeley.edu); (6) Armando Fox, Computer Science Division, University of California, Berkeley, Berkeley, USA (fox@berkeley.edu). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv [5] By the time the experiment was carried out, COVID-19 restrictions in force in Andaluc´ıa allowed students to be in the same classroom but wearing masks.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Inside a Study on Gender Bias in Remote Pair Programming

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

11 Proven Solutions to Common GitHub Copilot Problems

How Does Perceived Gender Impact Pair Programming Dynamics?

Researching Coding Bias with Twincode

What We Learned From 92 Students in a Spanish Software Class

Can Gendered Avatars Affect Teamwork in Coding?

Gender Perception Doesn’t Sway Pair Programming Outcomes

11 Proven Solutions to Common GitHub Copilot Problems

How Does Perceived Gender Impact Pair Programming Dynamics?

Researching Coding Bias with Twincode

What We Learned From 92 Students in a Spanish Software Class

Can Gendered Avatars Affect Teamwork in Coding?

Gender Perception Doesn’t Sway Pair Programming Outcomes

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps