Consistency is Key: The Beginner’s Guide to Predictive Technical Interviews

By Zach van Schouwen, Director of Interview Content.

Technical interviewing is something that every company does when hiring software engineers, but often with mixed results. Mis-hires are unfortunately common. Team members tasked with conducting technical interviews will likely use an approach that they are comfortable with, but that others don’t have visibility into and hasn’t been tested for ability to predict performance.

What could go wrong? The interviewer may:

• Assume a candidate knows the criteria for success (when they don’t)
• Inadvertently overvalue the answer to an irrelevant question
• Behave or communicate differently with each candidate

These are just a few actions that can cause inconsistency in the interview. This will reduce signal for the factors most capable of predicting potential — or fail to measure them at all. Good candidates may end up being rejected, when they could have added significant value to the organization.

Creating consistency in variables like questions, a candidate’s background knowledge, and the interviewer’s communication style can go a long way to ensuring that the only variable in the interview is the one you’re trying to evaluate — the candidate.

In my role as Director of Interview Content at Karat, I work with our clients to pinpoint exactly which competencies will be critical for success in each role. This guides us in choosing the right questions to ask. An equal amount of work goes into deciding how to ask them in a way that will most accurately determine the potential of the candidate.

To accurately identify the candidates most likely to succeed in software engineering roles, we control for dozens if not hundreds of variables in the first-round technical interview. Below are four ways you can dramatically impact the ability of technical interviews to predict onsite and on the job performance.

Decrease the risk of false negatives by making criteria for success available to the interviewer and candidate.

Simone Biles can’t keep winning gold medals if she doesn’t know what success looks like according to those that hand out medals. Interviewing for a job that requires specific technical skills isn’t really that much different.

Candidates and those on the interview loop should be fully aware of the skills and competencies that are relevant to the role. These should be agreed upon across the organization and measurable in the interview process. For example, do you care about the optimality of candidates’ solutions in terms of time/space complexity? If so, how are you measuring this skill, and how predictive is it of future performance on the job? When interviewers and candidates start with the same knowledge of which criteria are determining factors for success, candidates perform better in the interview. This decreases the risk of false negatives.

To make sure that the criteria are clear, establish what success looks like and how it will be measured. This should be communicated and agreed upon by all team members. Categorize certain behaviors or skills and then use descriptive statements to identify what beginner, intermediate, and advanced work looks like.

Below is a sample question with scoring guide. For context, this is the third question in a series of questions.

The candidate and interviewer see the question, while clarifications and scoring categories are seen by the interviewer.

This question provides a clear background and instructions for the candidate. Likewise, it includes clear scoring criteria for the interviewer. Any interviewer, hiring manager, or recruiter reviewing the candidate’s results from the interview would know exactly what “beginner” meant as opposed to “advanced.”

Criteria for success should be based on tasks that are relevant to the job.

In 2017, Software Engineer and Data Scientist Tikhon Jelvis took to Quora to answer the question “what is the future of software engineering interviews?” He said, “I don’t know what will come next, but I know what should come next: carefully calibrated and standardized work-sample tests.”

Technical interviews should be predictive of performance at the onsite and on the job. Therefore, technical interviews must rely on performance demonstrated in relevant, real-life problems and challenges. If your company seems to experience a high percentage of false negatives (i.e. rejecting candidates after the technical screen) only to find out later that the candidates went to a high-bar competitor, your technical interviews may not be accurately evaluating skills that are relevant to the job.

For example, I’ve seen good candidates rejected from a Java developer role after not answering the following question properly: “given the (x, y, z) coordinates of four points, determine if they all lie in a single plane.” This might be relevant to a few game designers, but not a typical back-end developer.

Be rigorous in the process of determining which questions and criteria are truly relevant to the role you’re hiring for. Those that are irrelevant could cause you to reject an otherwise amazing candidate.

Communication during the interview matters just as much as asking the right questions.

What does verbal and nonverbal communication have to do with consistency and predictiveness, you might ask? Communication is a critically important variable to control for in the technical interview. Interviews are nerve-wracking. An interviewer’s communication and behavior can exacerbate the level of stress a candidate experiences and stifle their ability to problem solve and think creatively. This isn’t good for the company or the candidate. It’s very likely that interviewer communication will be inconsistent across your organization if those tasked with interviewing haven’t been trained on it.

Below are a few examples of communication styles interviewers should avoid and instead, those that they should engage in.

Examples of three communication styles interviewers should avoid and three to engage in.

Thinking carefully about in-interview communication and behavior isn’t enough. The team involved in technical interviewing should practice role playing, with clear guidelines, over an extended time period in order to really get it right. At Karat, interview engineers never really stop practicing and improving — they are required to complete 25 hours of hands-on practice prior to conducting technical interviews on behalf of clients. Over time, they are continuously quality checked.

Accountability is a must.

Creating consistent questions, defining your success criteria (and being transparent about it), and enforcing proper interviewer communication are all important, but if these standards are not evaluated in a continuous feedback loop, the interviews risk becoming inconsistent and will fail to be predictive. Less experienced interviewers need to be equipped with structured and data-informed best practices to conduct technical interviews. Even strong interviewers can develop bad habits if not checked.

In order to implement a technical interview that remains consistent and predictive over time, technical interviews should be thoroughly documented or recorded (with permission, of course) and evaluated by a more experienced interviewer to check for aberrations that lead to inconsistency.

Over time, this feedback helps interviewers become significantly more consistent and less prone to these aberrations — and many are even able to avoid them entirely.

Final thoughts

A consistent technical interview is the only way to ensure that a candidate is truly being evaluated for their potential and produce an accurate assessment of that potential to the hiring team. I’ve personally seen inconsistent and irrelevant interviews result in the rejection of too many great candidates (an action that disproportionately affects underrepresented minorities). At Karat, my job is to make sure that doesn’t happen. This dedicated advocacy has obvious benefits for job candidates, but it also helps the hiring company as they find that more candidates are landing in the right role and becoming more productive more quickly — and having great interview experiences that get them there.

To learn more about Karat, please visit Karat.com.