From Gut Feeling to Data-Driven: Navigating Bias in the Tech Interviews

Hiring the right talent is a crucial part of building a healthy organization. Obviously, professional and motivated people deliver bigger outcomes. Furthermore, a suboptimal hiring decision will result in a loss of money. In 2019, the US Small Business Administration estimated the cost of a bad hiring decision at between 1.25 and 1.4 of the yearly salary[1]. Even for experienced leaders, conducting interviews can be difficult. Based on a few hours of interviews, you need to find out if a person is a good fit. The stakes are high. Yet, managers often receive little training on hiring.

On the other side, preparing for the interview has become an industry. Tons of books written on how to pass an interview. Online resources offer mock interviews. There are various templates and consulting services available to prepare a CV. Candidates are eager to receive a good job and are motivated to present themselves in the best possible light. However, hiring managers are interested in knowing the candidate's true achievements and skill levels.

This puts even greater demand on the hiring process. Only by digging deeper than the polished CV and well-rehearsed answers can we evaluate if the candidate will be successful. A well-structured interview will help to avoid two types of mistakes:

• Hiring someone who looks good on paper but who won't be able to do the job.
• Rejecting candidates who are great for the job but are having trouble navigating the interview.

Interview questions don't tell you everything about a candidate's ability to do the job. On the job, you don't design a distributed system in one hour on a whiteboard. You do not solve complex algorithmic problems every day. These questions are useful for evaluating technical skills, but they also show how the candidate behaves. Are they able to think through the problem? Did they consider alternatives for the solution? Can the candidate communicate ideas clearly?

The candidate's behavior might be mixed with the interviewer's interpretation of it. Statements like "clear communication" are ambiguous and might be interpreted in different ways. The candidate may address questions accurately and to the point, but their pace of speech is a bit too fast or too slow due to anxiety. The candidate might not be a native speaker, and their accent might be hard for the interviewer. Things like that are easy to label as "not clear communication."

What is Bias?

The success of the hiring decision depends on what you ask and how you interpret the answers. The incomplete information creates a challenge for the hiring managers. The decision-making process might be influenced by previous experience, the interviewer's background, and various non-rational factors. This where bias can sneak in into the decision-making.

A bias is a tendency, inclination, or prejudice toward or against something or someone.

Practically, it skews judgment in a certain direction. Daniel Kahneman gives a simple example of bias in his famous 'Think Fast and Slow': “Julie is a graduating senior at a university. She read fluently at the age of 4. What is her GPA score?”

With no information about a student, it would be reasonable to guess an average GPA of 3.3. However, the piece of barely relevant information that she read fluently at the age of 4 made people guess higher scores in Kahneman's experiments. "Julie" is a made-up example, you might argue. Yet, I heard from real interviewers something similar to, "This candidate's CV is in LaTeX; they must be incredibly smart".

Knowing more about bias and learning different types of cognitive biases[2] might help, but it won't be enough. We can compare cognitive bias to an optical illusion—Knowing it's an illusion doesn't help us see reality.

You can't rewire yourself to be free of bias, but you can design a hiring process that accounts for it. In my research on bias in interviews, a came across a great example of how symphonic orchestras changed their hiring approach.

In 1970, only about 5% of staff in the top 5 symphonic orchestras in The U.S. were women, and the percentage hasn’t changed much since 1940. By 1997, about 25% of musicians in these orchestras were women. All these years, female musicians graduated from the same music schools and put in the same amount of effort. Yet, they didn't get the opportunity to perform in top orchestras. Even though the main focus of auditions was to evaluate a musician's ability to perform, other factors influenced the jury's decision.

According to the research[3], the main contributing factor to the increase in the female share of new hires was the introduction of “blind” auditions. In the “blind” setting, candidates were playing behind a screen. Hiding the identity of the player from the jury significantly increased the chances for female candidates to advance in the hiring process.

Before the interview

A well-designed hiring process starts with the profile of a perfect candidate. What skills and attitudes must they have to be successful in their role? Responsive design and accessibility are crucial for frontend engineers, but they do not play a big role for the backend. On top of skills, you want to add attitudes that reflect the culture of your team. Is collaboration crucial for your team? Maybe the company has in-house technologies, and it's crucial for the candidate to be able to learn new things. The goal of having a written candidate profile is to reduce variation in the decision-making process. It helps the hiring manager evaluate candidates only based on the skills and attitudes listed in the profile.

It's important to derive questions from the attitudes you defined in the candidate profile. Think of sufficient evidence for each attitude and write questions that will help you drive the conversation with a candidate. Here are some examples of attitudes and questions:

During the interview

Your job as an interviewer is to collect sufficient information for each attitude. Questions in my example a deliberately broad, they open the conversation and provide opportunities to follow up on the candidate's experience. Vague and impersonal answers might get in the way, but you need to dig into details to assess the candidate's experience.

— HM: Tell me about a time when your work was blocked by someone else's task?

— C: My team was blocked by another team. We needed a change in the API of their service.

This is a vague answer, it doesn't give you much information. You might need to steer the conversation toward the candidate's actions.

— HM: Tell me more. How did you handle this situation?

— C: We organized a meeting and discussed all the details.

Still not enough. Their contribution isn't clear, you don't know who's "we".

— HM: I understand there were other people involved. Can you please describe the steps you took to unblock your team?

— I: I went to the code base of the other team and checked if they had data that my team needed. I also briefed my manager on why we need this new field and shared the results of my research. Then, together, we reached out to another team and agreed on the timelines.

This answer might be sufficient, depending on the level of seniority you're looking for. Various frameworks exist for answering interview questions: STAR, PAR, CARL. Not knowing them doesn't make the candidate a bad fit for the job. As an interviewer, you need to search for sufficient evidence from a candidate. You probe each attitude with a question and see if candidates can provide detailed examples of when their behaviors were aligned with the attitude you're assessing.

After the interview

A final hire/no-hire is rarely made immediately after the interview. Interviewers might have a meeting to discuss candidates and the results of each round. Information might be lost or changed dramatically between the time of the interview and the final decision. To avoid this, each interviewer should write down their decision and supporting evidence soon after the interview and before the discussion with the other participants.

It's important that each interviewer makes their own decision based on the information they gather. The worst thing that could happen is a group of interviewers who aren't sure what they want to do, sitting in the room and discussing how to proceed. An interviewer might prioritize their peers' opinions about the candidate over their own evidence. The group might agree with the strong opinions in the room and end up in groupthink[4].

As a hiring manager, make sure all interviewers list their observations in the scorecards. What evidence did they find about skills and attitudes during the interview? Pay particular attention to the language on the scorecards, and make a distinction between observations and interpretations. You see things like "that was a weak candidate" or "they aren't good enough for the role", probably the jump from skills to the candidate's personality happened somewhere in the process. Ask the interviewer what insights led to this conclusion. Remind the them that we don’t judge people; we assess their fit for the job based on competencies critical to the role.

Conclusion

Interviews happen as conversations between people, and as with every conversation, you might like it or not. However, personal preferences, which can be biased, shouldn't be the main reason for hiring someone. The goal of the interview is to gather evidence of the candidate’s ability to perform the job. How you conduct the interview and the process you follow make the difference between “I kinda like them” and an informed decision. Remember that not seeing evidence doesn’t mean its absence. Even the best candidates on their worst day might have a poor interview outcome. You never know what a candidate is going through. Design your process, be humble, and treat people with respect.