Predicting the Success of Pair Programming Interviews

03 February 2020

Over the last 4 years I’ve interviewed hundreds of engineers, first as a manager at a large company and then as the head of engineering at a startup. I believe in and heavily use pair programming interviews, in which the candidate and interviewer work through a small programming problem together.

After conducting over 50 such interviews myself over the last few months, I’ve found that the success of a candidate can be predicted from 2 questions:

• What is the first line of code they write?
• How often do they run the program?

In each case, the way to succeed is obvious:

• Candidates that start with the main function and proceed from there are much more likely to finish the problem and produce higher quality code than candidates who try to design the solution upfront or start with OO modeling.
• Candidates who frequently run the program, either through unit tests or directly, spend much less time debugging than candidates who do not.

I’ve found that these two questions are much more predictive of success in the interview than years in industry or academic training. Let’s go into some details.

Interview format and problem details

Our interview format is simple: the candidate and I sit together and work through the problem. We encourage candidates to bring their own laptops and use the programming language and IDE of their own choice. At the beginning of the interview, I verbally explain the problem and provide them with a 2-page PDF that goes into more detail.

Our current problem is a text-based stock market simulation. The program tracks the performance of some investors’ stock portfolios over time while receiving text commands like INITIAL PRICE GOOGL $100, FRED BUY 5 GOOGL and GOOGL UP $3. At the end of the input stream, the program must print the costs and returns for each investor. The problem can be solved with 2 hashmaps and terse implementations regularly come in under 200 lines of code.

While the candidate does most of the typing, I try to be as helpful as possible without guiding the design or overall approach – pointing out typos, asking about cases the candidate missed, that sort of thing.

Run your code often and don’t start in the middle

After reading the problem and talking it over with me for a moment, candidates often open a file and write something like:

def initial_price(ticker, price)

def buy(investor, ticker, shares)

Or:

class Investor
  def buy(ticker, shares)

Starting in the middle of the problem this way almost never works, for several reasons.

First, the initial design is always wrong and usually is so far off it needs to be thrown out later on; the candidate burns valuable time.

Second, this code is impossible to exercise with the actual inputs the program will handle, since it relies on a main function that hasn’t been written yet. This means the candidate either doesn’t test the code until much later or they burn even more time by writing unit tests around a totally incorrect design.

Third, candidates can get so engrossed in domain modeling they drift away from actually solving the problem, burning even more time.

Candidates who start with the main function, on the other hand, are forced to immediately confront the design decisions that actually matter in this problem: how do they separate parsing and execution, and where do they store state?

How much does this matter?

In an interview a few months ago, a candidate we’ll call Fred, with a CS bachelors from an elite university and 3 years of professional experience, started this interview by trying to write an Investor class. He floundered for 20 minutes before I suggested he start with the input handler. He didn’t run the program until the end and wrote no tests. When he did start running the program, there was an order-of-arguments bug in the middle of the code that took 15 minutes to find.

The next day I interviewed another candidate, James, who majored in business and segued into software 2 years prior via a bootcamp. James started with the main function and drove the program out by invoking it every 30 seconds and liberally using print statements. He finished easily.

Here’s the thing: Fred was by most measures a much better programmer than James (although neither candidate advanced to the next round). James struggled to understand how to use a double-nested hash, repeated parsing logic multiple times and generally needed a lot of help. Fred had an excellent command of his development environment and clearly understood the impact of his changes before he made them. Nonetheless, he couldn’t overcome the twin handicaps of inside-out design flow and not running his code.

Make things easy on yourself

What does this have to do with programming outside of interviews? I’ve concluded that there are only two ways to distinguish yourself in engineering: you either have to be much, much smarter and more talented than your peers, or you have to find ways to make the job easier.

If you try to write a program from the inside out and never run it until the end, you are effectively placing a huge bet on your own intelligence, understanding of the problem and ability to hold it all in your head. And if you screw up halfway through, you won’t realize it until the end.

The best engineers I’ve worked with never, ever do this. Instead, they use tools – tests, type systems and so on – and processes – like outside-in TDD – to minimize the amount of ‘holding it in your head’ they have to do. The best way to be a great engineer isn’t to be the smartest person in the room; it’s to arrange your work so that being super smart isn’t required at all.