An Interviewer’s Perspective

Photo by Andrew Neel on Unsplash

You’re nervous. The phone interview is about to start. This could be your one chance to get the job you’ve wanted - at least until next year. It all comes down to this – 45 minutes of proving your problem solving and coding skills.

Suddenly the phone rings.

You pick up and feel your heart beating. After some quick pleasantries, the interviewer pastes the first question into your shared coding environment, and starts explaining the problem.

Your mind goes blank. The nerves have got to you, and you just can’t think clearly under the pressure you’ve put yourself under.

You keep checking the time. 5 minutes have gone by.

Silence.

You’re aware that the interviewer has been waiting for you to say something. You try to fill the silence with your thoughts, but this just disrupts your thought process.

You check the time again. 20 minutes have gone by.

You panic and before you know it, the interviewer says time is up and you have an opportunity to ask them questions.

Your heart sinks.

There’s no chance you’re getting to the next stage. But you muster up a few questions and ask them half-heartedly anyway.

The call ends and you get a rejection email a few days later.

Rationally, I knew I was good enough. But after going through this horrible process a few times, I started to feel like I wasn’t.

I could tell myself that I knew how to code better than most. That I had built websites and apps for years, some used by thousands of people. That I was doing well in my Computer Science degree at Oxford. But emotionally, the pain of a string of rejections still got to me.

I channeled this pain into a determination to get good at coding interviews. I bought a whiteboard, some markers and a copy of “Cracking the Coding Interview”.

In a couple of weeks, I went through every question in the book. For each one, I solved it with code on the whiteboard, while vocalizing my thought process, then typed up the code into the IDE on my laptop. I wrote some test cases and made sure my solution worked. I checked the solution at the back of the book too. For every mistake I made, from small syntax ones to not using the right algorithm, I wrote it in a mistake document. Before starting the next question, I reviewed all the mistakes.

The next round of interviews went well. I got multiple offers and decided to join Improbable, a London tech startup.

After a year at Improbable, I decided to move on for various reasons. I had to do coding interviews again but it had been a while so I needed more practice. I repeated the same process on questions from LeetCode and HackerRank. The LeetCode premium subscription gave me access to a huge question bank categorized by the companies that asked them and sorted by frequency. This was an awesome way to focus my practice on the most frequently asked questions.

After this round of interviews, I accepted an offer to join Facebook.

I’ve now been at Facebook for over 2 years and find myself on the other end of the phone, interviewing candidates for software engineering roles. I regularly see people struggle with their nerves and make the same mistakes I did. I know that this is not a reflection of their ability as a software engineer, or a particularly good indicator of their future job performance.

So why do we still do it?

I know that in my case, my ability to perform well on the job did not change as a result of me learning to pass coding interviews. But the fact that I was able to learn to pass coding interviews proved that I had the raw technical skills necessary for the job.

If someone can pass the coding interview, they are highly likely to have the technical skills to succeed on the job – they would be a good hire. And if someone fails the coding interview, there’s still a pretty good chance they would be a good hire.

Let’s unpack this a bit more. Essentially, an interview is a method of extracting signal from a candidate that can be used to predict whether or not they would succeed on the job. Like any prediction task involving classification, sometimes you’ll be right and sometimes you’ll be wrong.

When the amount of signal you can extract is limited due to time constraints on both you and the candidate, the accuracy of your assessment will be lower and you’ll make more mistakes.

When you’re wrong about someone being a good hire, it’s a false positive. And when you’re wrong about someone being a bad hire, it’s a false negative.

The fact is, false positives are a far bigger problem for companies than false negatives. The cost of hiring the wrong person is extremely high – new hires are initially a net loss to companies, as they have to go through an expensive ramp up and on-boarding period where they are not productive, take time away from others, and still get paid a salary. If at the end of all of this, you have to go through the process of firing a candidate, you never get back this cost. If firing an employee is difficult due to employment laws in the country, the cost can rack up.

On the other hand, rejecting a good candidate costs very little. And as the volume of good candidates who apply increases, the cost of rejecting one decreases.

In any binary classification task, you’ll have a false positive rate and a false negative rate. Then you can change your classification threshold – the point at which you predict someone as positive rather than negative – to optimize these ratios.

In this case, the false positive rate is:

(# of bad candidates you hire) / (# of people you hire)

The false negative rate is:

(# of good candidates you reject) / (# of people you reject)

When you increase your threshold (hire bar), you decrease the false positive rate at the cost of increasing your false negative rate. This means that companies who have a lot of applicants, and invest a lot in new hires are heavily incentivized to raise their thresholds.

This system usually works well for companies – which is why it hasn’t changed much. But it doesn’t work for individual candidates, as it just treats them as a number in a large applicant pool, and can make being lucky important.

Since interview performance doesn’t correlate perfectly with job performance, companies will often reject candidates who would be more successful on the job than candidates they make offers to.

This is particular the case for coding interviews. Candidates who spend more time honing their skills in the domain of solving traditional coding problems, and less time building real things in the real world might have an advantage.

Just look at this tweet by the founder of Homebrew, Max Howell:

So, how could the system be improved?

The purpose of the interview is to extract enough signal from a candidate to predict whether or not they would be successful on the job. If you had perfectly accurate predictions you could accept everyone who would be successful on the job, which would be great for companies and candidates.

An almost perfectly accurate method would be to just give candidates the job for some fixed period of time, and if they are successful, offer them a permanent job. This is what an internship is – an extended interview. The problem is that this is a risky investment for the company. Which is why internship offers rely on coding interviews too.

In practice, companies have a limited amount of time and resources to extract signal from candidates. And candidates often have a limited amount of time to do interviews.

Within these time constraints, companies want to collect signal on a few key areas that are necessary to succeed as a software engineer.

As an interviewer, I want to see that a candidate has knowledge of basic data structures and algorithms. That they can use this knowledge to come up with an algorithmic solution to a problem, talking through other approaches and tradeoffs. I want to see them code up the solution in a programming language, reason about the time and space complexity, walk through their code and test it.

So the general format of a coding interview does make sense, if you only have 45 minutes.

Fixing The Coding Interview

That being said, the whiteboard is just unnecessary. If you haven’t coded on a whiteboard it can feel a bit weird and throw you off. So if a candidate prefers using a keyboard, companies should just let them. You don’t want to reject a candidate because they were not comfortable coding on a whiteboard.

Another thing to get right as an interviewer is the choice of problem. You don’t want to choose a problem that requires a giant creative leap or domain specific knowledge in some obscure area of maths or computer science that is not broadly applicable. Ideally, the candidate should be able to reason through the problem logically and iteratively arrive at the optimal solution. You don’t want to reject a candidate because they didn’t have the necessary eureka moment in a 45 minute interview.

Finally, it’s important to get the candidate to relax. It’s hard to be logical and methodical when your mind is racing and under pressure. Settling in to the interview with a few quick questions about their favorite past projects can be a good way to calm the nerves. You don’t want to reject a candidate because they find time-pressured coding stressful.

But beyond this, it’s hard to increase the accuracy of your signal from an interview without relaxing the time constraints.

Getting More Signal

Some companies will ask candidates to do a take-home assignment to remove the time pressure and allow for a more involved coding problem that provides more signal.

Some will collect more signal by having many stages to the interview process, expecting candidates to take multiple days off work.

And some will conduct a greater variety of interviews, testing more practical skills like navigating a large codebase or building a React frontend interacting with some public API.

Although these methods provide greater signal, allowing companies to make more offers to good candidates, they can also demand too much of candidates’ time, which leads to an unpleasant interview experience. The problem is that candidates apply to multiple companies at a time, and it’s just not feasible for each of them to have extremely time-intensive interview processes.

Interviewing as a Service

One solution to this is to unify the coding interview process across multiple companies. An independent coding interview service could spend 5-10x more time interviewing candidates than any individual company could, allowing them to get much clearer signal on candidates’ technical ability, reducing the false negative rate while keeping the false positive rate low too. This would make for a much more efficient and pleasant experience for candidates, as they could avoid repeating the same interview process at multiple companies and take luck out of the equation. And it would free up companies’ resources to focus on the final non-technical interviews like team and culture fit. Companies like Triplebyte are already trying to do this. But they still seem to act as more of a screening service than a full replacement for the technical interviews.

I’d imagine the main challenge of replacing larger chunks of companies’ own technical interview process with a more accurate, unified and independent version is to gain their trust in the process whilst allowing customizability with respect to assessment of specific technical skills and hire bars.

Final Thoughts

As a candidate, it’s important to remember that a 45 minute coding interview is not an accurate assessment of whether you’d be successful on the job. The high false negative rate means you definitely shouldn’t feel bad about being rejected. A rejection often just means you need more practice displaying the technical ability you already have in an artificial, time-pressured environment.