Technical Interviews: What is the Best Way to Measure a Candidate's Skill?

How can CTOs measure a candidate's skill?

This is a huge part of technical recruiting because CTOs want to filter for the best candidates with the least amount of time and effort for both parties.

Should one do live coding or a take-home assessment?

Perhaps one should stick to whiteboard problem solving or have the candidate analyze a PR?

Some even argue that in today’s market - we should skip challenges entirely! We asked experienced CTOs from the Alphalist CTO network and this is what we learned.

TLDR: Ideal Interview Testing According to Seniority

• Junior Role - Take Home Assignment and Code Analysis
• Mid-level Role - Live Coding. Whiteboarding and Portfolio'
• Senior Role - Code Analysis, Live Coding, Whiteboarding, and Portfolio

4 Different Ways to Assess a Candidate's Technical Skills

Take-Home Assessment

A Take Home Assessment is when the candidate is assigned a coding task to do at home.

Benefits of a Take-Home Assessment (AKA homework)

Take-Home Tests are a way to allow a candidate to prove their ability in a non-pressurized environment.

“Take-Home Tests are the less stressful alternative to live-code tests because the option to do the assessment in private helps many people who would otherwise be paralyzed in an interview setting. “We do take-home assignments for junior & mid-level roles. Those are the candidates who tend to get most nervous during live coding,” says Steffen Wenz.

Take-Home Tests are also ideal for junior roles where they give those without much of a portfolio a chance to prove themselves and their ability to adapt to new technologies.

One CTO relates how he offers a take-home challenge to give those who don’t yet match job requirements a chance to prove that they can learn quickly. Another benefit of Take-Home Assignments is that they are asynchronous - which means no one’s schedule is tied up and also scales well so you can offer the same task to multiple people at the same time. (A one-to-many relationship - but you will need to change them of course!)

Cons of using Take-Home Assessments in Technical Recruiting

Many companies no longer offer to Take Home Assessments in today’s competitive market due to the fact that candidates would rather not -or cannot- do ‘homework’ in their own time.

First of all, Take Home Assessments discriminates against candidates who cannot spend multiple hours on an exercise on top of their current positions - reducing diversity in hiring and the talent pool.

“We stopped doing take-home challenges recently as we got some feedback that it was selecting a very specific type of candidate that can spend multiple hours on an exercise on top of their current positions (i.e. not moms and dads for instance).” - CTO of European unicorn

Secondly, candidates may drop out if they get other offers where they don't have to do any extra challenges in their own time. This is particularly true for mid-senior level roles.

“My experience from the ANZ market is that lead and senior developers (sometimes even mid-level ones) will not consider a role that requires a technical test, let alone a take-home one. It simply has been a candidates' market, and if one has 3-4 similar roles to consider, they would usually go with the ones that require less effort from their end. It is also a matter of not having lengthy hiring processes as most candidates expect 2-3 weeks from application to offer. There are of course exceptions and it's even more critical 'selling' the role and the company to the candidate.” - Loucas Gatzoulis, Chief Technology Officer at Education Perfect -Australia

Furthermore, others argue that coding style syntax can be taught and what matters most is abstract thinking (better tested in White Board Interview) and a deep understanding of how to write a line of code (better conveyed in a live code setting).

Best Practices for Offering Take-Home Assessments

Who is a Take-Home Assessment Ideal for? A Take Home Assessment is ideal for junior roles. They do not have a portfolio and would be able to prove themselves by showing how fast they learn a new stack. It won’t work well for senior roles where you want to hear their through process more.

What is the Best Way for a CTO to conduct a Take-Home Assessment?

First of all, make sure the candidate knows what is important to them. For example, if testing is important to you - let them know. It is also best to limit your take-home assessment - a 2-hour time frame is common. Secondly, choose questions that can't just copy/paste from the web…e.g. “what if we would ask you to do this and that”, or “How would you implement XXX”

What should you centre your Take Home Assessment around? It is up to you. Some CTOs like, Daniel Bartholomae of optilyz, choose to center the assessment around an open-source project while another CTO uses it as an opportunity for the candidate to showcase building a proof of concept using the unfamiliar technology of the company’s stack.

Live Coding

Live coding is a form of pair programming in which candidates and interviewers tackle a coding challenge together.

Benefits of using Live Coding in a Technical Interview

Live coding is a much faster and more efficient way to test candidates' abilities while giving the interviewer additional insight into the candidate’s thought process and communication abilities. Live coding requires an hour of time from both parties, as opposed to a take-home task which requires up to 4.

It also allows the interviewer to get an idea of the thought process of the candidate by asking about their motivations as they code. Sebastian Mach, CTO of Byrd, says he also prefers live coding as it shows how well a candidate would work with a co-worker and also tests the candidate’s communication skills - which is particularly important for senior-level roles.

Disadvantages of using Live Coding in a Technical Interview

However, live coding does not always bring out the best work of all candidates due to its pressurised nature, and its limited scope. Many people - especially juniors- do not do well under pressure. Furthermore, as Daniel Seif relates, how when he coded under the watchful eyes of a remote interviewer, it not only felt awkward but “really only checked how I would implement the code for a solution I was being dictated”. Another CTO relates that “I’m somewhat against the live coding challenge due to the pressure it would create on the candidates and they strongly favour people who succeed under pressure and alienate those who would feel uncomfortable in these scenarios.”

Best Practices for Conducting a Live Code Interview

What type of role is most suited for a Live Code Interview? Live Code interviews work well for mid-senior roles where you want to hear their through process more. However, it's better to not use it in a junior role where it might put too much pressure on junior applicants.

What is the Best Way for a CTO to conduct a Live Code interview? First of all, try to reduce the stress as much as possible - by making the candidate feel comfortable with specially trained interviewers and choice of tools. Training interviewers is important. ”IMO, the interviewer(s) need to be trained to make the live coding exercise not stressful (unless that’s what you’re trying to assess obviously).” says the CTO of a European unicorn.

As Jacob Kaplan-Moss relates, the interviewer needs to be skilled at both pair programming (collaborating - after all, they are not just sitting there watching) and interviewing (need to get insight while maintaining psychological safety). Furthermore, give the candidate as much flexibility as possible. For example, you can let them choose their own IDE.

This is strongly advocated by Darío Blanco, Chief Technology Officer at KIRINUS Digital GmbH, who says “I strongly think that selection processes should be as flexible as possible to allow the candidate to perform without artificial pressure that is never really replicated in the work environment (who codes without an IDE of their choice after all?)”

What is a good task for CTO to give a candidate in a live code environment? It all depends on one what you are looking for and what is important to you. For example, one CTO wants candidates to get a glimpse of the actual code they might work on within the company so he uses a collaborative code editor to work on a modified version of their actual codebase.

White Board Problem Solving

One uses a whiteboard interview to assess the candidate's grasp of concepts by having the candidate walk the interviewer through the technical decisions they would make regarding algorithms, data structures, and general architecture in a particular problem. This often leads to a discussion with the interviewer. Popular in the days before the internet, it is still done remotely today.

Benefits of White Board Problem Solving Problem Interviews

Whiteboard problem-solving interviews allow the CTO (or interviewer) to test high-level thinking without getting lost in the nitty-gritty of syntax. It brings out the background knowledge of the candidate and shows their problem-solving skills.

It also shows how well the candidate can explain their decisions. “The communication aspect of presenting a problem and discussing it with the interviewer is much more important than a particular programming style for us. “ says Daniel Seif. He once even hired someone with only basic knowledge about their programming language ( they were proficient in another), simply due to their attitude and motivation and their problem-solving skills.

By thinking beyond the code, companies have seen success with the whiteboard method as it allows them to hire on potential as well. “We invested heavily in having good problem-solving/whiteboarding challenges that test for all the behaviours and skills we had previously covered in a code assessment,” says Elmar Weber, CTO at Amboss.

“Coding style and adherence to our standards is something that is always easier to transport to new hires compared to mindset and ways of solving problems. This widened our talent pool and helped us to put more qualified people into a later stage and ultimately get them hired.”

Disadvantages of Using a Whiteboard-style Interview

The primary disadvantage is that it's not the most popular method for developers. Some devs hate being whiteboarded. There is even a black list of companies that whiteboard. Why do devs hate it? Some devs fear the amount of prep they would have to do before a whiteboard interview - memorising many algorithms and data structures. Or perhaps they like coding - not necessarily explaining themselves.

This might be a good thing though if you need someone who has these skills as it will help you weed out the good communicators. However, even so - it does discriminate against those who don’t interview and aren’t confident expressing their thoughts to strangers under pressure. The candidate might be able to write essays on why they would choose an algorithm but when it comes to interviews they freeze up.

Best Practices for Conducting a White Board Interview

For what type of roles is a Whiteboard Interview most suitable? A whiteboard interview is ideal when hiring for mid- or senior-level roles where the candidate would need to make decisions on a high level.

What is the Best Way for a CTO to conduct a Whiteboard Interview?

As with any interview, candidate ease is a priority. “Reassure the candidate that the entire ‘interview’ is about the journey the discussion takes - not necessarily the final answer. In fact, there is no right or wrong answer.” says Elmar.

“If a visibly nervous candidate is interviewing for a mid-level position, make sure to reassure them by letting them know that it's okay to notify you when they are stuck or something is unclear. When it is a senior position, we expect them to manage this themselves and actually at one point take over and steer the conversation.”

What kind of questions should you ask in a Whiteboard Interview? Whiteboard interviews usually center around a high-level business question. Elmar usually chooses a problem connected to a challenge the company dealt with in the past or connected in a fun way to what the company does.

They have different challenges for data, frontend, and backend roles. It usually focuses on system design, “We drill into different areas like algorithms, data structures, architecture depending on the role, level of experience and how the conversation goes” says Elmar.

Code Analysis (AKA ‘Reverse Coding Assignment’) e.g. PR

In a Code Analysis Interview, you assess the candidate by talking through how they would improve the code in either a real Pull Request / Component taken from your stack

Benefits of Using Code Analysis In A Technical Interview

PRs are the most time-efficient method as they require the least prep work. It allows the company to get an insight into the candidate's code quality standards and their testing philosophy.

It is also the least stressful for the candidate and allows junior candidates to showcase their potential. As testing is important to him, Darío Blanco Iturriaga, CTO at KIRINUS Digital, always uses code analysis to see how the candidate would test the code.

Best Practices for using Code Analysis in your Hiring Process:

When should a candidate be assessed using code analysis? Code analysis type interviews are most suitable when you are hiring for very junior or very senior positions.

 Jacob Kaplan-Moss suggests using ‘reverse coding’ for junior roles as it reveals their potential in a way that's less intimidating than live coding and easier for them to do than a Take Home assignment. He also recommends it for very senior roles whose roles would require mentoring and reviewing a lot of PRs! A PR test like this will test the broadness of their analysis skills and also the way they communicate the critique.

The Best Method For Assessing Candidate Technical Skills (according to experienced CTOs)

After speaking with CTOs from a CTO community for experienced CTOs, we saw that most CTOs offer options to the candidates or use a mix of techniques to assess candidate level.

One CTO even said he only conducts coding challenges for mid-level positions, and that is only when the technical interview came out vague. (He uses an internship partnership with a local university as a feeder for his junior positions though).

Offer Candidates a Choice

In this candidate market, it is best to offer options so you can cater to all types.

"We will offer two routes: One with THA and one with Live assessment (coding and code review/bug fix modules, done live with pair programming with the interviewer). Both routes have some steps that are the same (eg. System Design). The reason we do it is that the market is getting very hot. We want to provide as many alternatives to candidates, so we remain competitve and attractive and engineers want to interview with us. -Thanassis Bantios, CTO of Beat - Greece

• Darío Blanco lets candidates choose between either a live pair programming session in any IDE of their choice, a Take Home that should not be more than 2h, or a live PR code review session (no programming involved but should give feedback about improvements in that code).
• H.R. lets the candidate choose between a 2-hour Take Home challenge or a 1-hour live coding session.
• Daniel Bartholomae offers a choice between a 1 hour live pair programming session or a 4-hour take-home test (+1 hour review). He does find most people prefer the Take Home Test.

Offer a Hybrid of Multiple Techniques

Sometimes, CTOs use a few methods to assess a candidate from all angles.

For example, although Elmar assesses candidates using the whiteboard method, he also asks them to bring some code they have written to verify if the person can code and how they code and structure projects.

This code could be a side project or open-source contribution. (If it's work they did at another company, the code is usually shown via Zoom so the actual code is never sent over). Most candidates have such code available but for those who don’t, Elmar uses the Code Analysis Method (PR) or as a last resort, he would assess the candidate’s coding skills using a Take Home Assignment.

Meanwhile, C. gives both a take-home test (3 questions, no code) and a version of live-code ( 2 or 3 exercises done together with the person using a shared editor) and has never received a complaint.

Summary: Pros and Cons of Each Technical Assessment Technique

Take Home Tests Pros and Cons:

• Pros
  • Less stressful.
  • Great for junior roles where the candidate does not have much of portfolio
  • Give the candidate the ability to prove they are a fast learner
  • Asynchronous
  • Scales well
• Cons:
  • Discriminates against candidates who cannot spend multiple hours on an exercise on top of their current positions.
  • Candidates may drop out if they get other offers where they don't have to do any extra challenges in their own time.
  • Time-consuming.
  • Tests coding style which can be taught.

Live Coding Pros and Cons:

• Pros:
  • It takes less time.
  • Assesses how well a candidate would work with a coworker
  • Tests the strength of the candidate’s communication skills
• Cons:
  • Discriminates against those who do not perform well under pressure.
  • Limited in scope to the implementation of just one solution.
  • Synchronous.

Pros and Cons of White Board Problem Solving Interviews

• Pros:
  • Can be done remotely.
  • Test high-level thinking without the distraction of language of syntax
  • Tests communication aspect of being presented a problem
• Cons:
  • Pushes away devs who hate being whiteboarded.
  • Discriminates against the bad interviewees.

Pros and Cons of PRs

• Pros:
  • Little prep work
  • Fast
  • Focusses on their standards for code quality
  • Showcases their testing philosophy
• Cons:
  • Does not show how they would create code

Thank you to technical leaders of the Alphalist CTO community for sharing their advice and experience both on Slack and for this article. Special thanks to Daniel Seif, Elmar Weber, Daniel Bartholomae, Loucas Gatzoulis, Dario Blanco Iturriaga, Thanassis Bantios,C.,Sebastian Mach, Steffen Wenz and others who contributed to this article.

You might also be interested in this article about how CTOs can optimize their technical recruiting process.

Co-published here.