Screen Engineers Better with a Debugging Roleplay

“Fill seats” was the mandate in spring 2015. Our budget was unknown, but our potential for growth was high. I think management was ready to expand our product footprint fast to take advantage of what traction we had.

This was the spark that ignited efforts to more than double the Engineering team at Localytics to nearly 70 engineers in less than a year.

Getting to “no”

At the height of my recruiting efforts I was hitting 15 phone screens a week on top of coffees, emails, interview loops, LinkedIn, AngelList, Greenhouse rubrics, and the rest of my duties as a Technical Director for the front end.

Phone screens became critical to our process given largely expanded candidate pipeline (we had 7 managers recruiting). If we didn’t rule out candidates, they’d eat up expensive engineering time in our interview loop, which could cripple the organization.

There are exits North and East

You should spend as little time as possible on these screening calls. The sooner you can decide if this is a “no” or “not a no”, the better. If there are no flags, advance them to the next step.

My technique for doing so is inspired by text-based role playing games. I first ask them if they’ve ever read a choose-your-own adventure book or played Dungeons & Dragons (which alone added a little fun to these chats).

I set the stage:

You own a successful product from toe to tip. You have access to every line of code and are responsible for the well-being of your business. The product is an analytics dashboard with some vivid data visualizations. Folks love it.

Here’s your scenario: A customer has contacted you for support. They say that they loaded up their dashboard and instead of a large graph they’re used to seeing at the top, the loading indicator spins for a while, then is replaced by a little red X after about 30 seconds.

It’s your job to debug and solve this problem, where do you start?

Now in my head, I have the solution played out, but I want this person to walk me through the steps they would take here. No cumbersome technology, just a engineering conversation about what the problem could be.

The problem as I imagined it ended up being some combination of a nested loop containing a query, and (in the fiction) this customer was generating data in a way that exposed some non-performant logic (like, this customer has 1000 of a particular record type where the usual customer has 50). The query would take too long and this API would respond with a timeout.

A great conversation would go something like this:

PK2: OK, so you’ve logged-in to the customer’s account and you can reproduce the problem. How do you proceed?

Engineer: Well, if there’s a little red X, I’m guessing I put that there. Hrm. Let’s open the inspector and take a look at the network tab.

PK2: OK, you open the inspector and look at the list of requests. One of them is red.

Engineer: Do I notice anything else about it? What’s the status code and response?

PK2: It says the request timed out. You notice that the URL and its parameters look like you’d expect.

Engineer: Oh, what are the parameters?

PK2: It looks like you have a dashboard ID, and timespan=90 (days)

Engineer: what happens if I change that to 1 day? does the graph load?

PK2: OK, you make the change and yes! It loads with one day of data and it loads very quickly.

…

You can imagine where it goes from there. We debug the API code, we find the offending method, we talk a bit about the data structure of the DB and they ball park a few solutions.

“we’re going to pass”

Given the exercise, I found myself filtering folks frequently. My advice here is to be direct, be human, and keep your mouth shut. “OK, based on what I’m hearing, you’re not quite at the level we’re looking for. If you were much stronger at X and Y, then I think you’d be an amazing candidate.”

After that, your job is to listen.

> go east

I encourage you to try this out. The example above was a facsimile of Localytic’s front-end product. I’m sure you could come up with a scenario based on your product and pluck from a hundred bugs that were closed over the past few months.

Let me know how it goes!