Interviewing odds and the Blackstone Ratio

..Interviews at many top tech companies seem to be set up such that there is a bias towards precision rather than recall..

NOTE/DISCLAIMER: the opinions I share in this article are my own, and not those of my employers. Please take them with the necessary grain of salt, as my (possibly, or probably, incorrect) interpretation. As always your mileage may vary.

Here I share an excerpt from Chapter 3 in Part 2 of Interview Poker, "The rules of the game." In this part of the book, I briefly introduce the concept of interviews being a somewhat random process with perhaps measurable odds, and then finally I bring in a familiar concept from the U.S. justice system to illustrate the point: the Blackstone Ratio.

Please enjoy. Hopefully if you are going through your own interview prep for some top tech company, it might inspire you to stay the course, accepting failures as not just opportunities to improve but also, quite possibly, simply a bad roll of the dice in a process that involves statistical error, inspiring you to try, try, try again.

"..The odds favor the house

A key point to remember about interviewing is that the odds favor the house. The process is designed to fail people—a lot—because of course there’s only so many jobs to go around and quite a lot of resumes coming in, and the cost of a bad hire to the team is high. In a 2014 article, Quartz, Forbes, and others, quoted Google as having received around 3 million applicants in a year, while the amount they reported hiring was somewhere around 7000, or roughly 0.2% [1]. Meanwhile, the acceptance rate for undergraduates at top universities like Harvard and Stanford is at least an order of magnitude higher. At that rate, you could get accepted into Stanford around 45 times before landing a job at Google, for instance.

If I’ve sufficiently discouraged you from trying or reading any further, fret not; those numbers can be quite deceiving. They assume a lot of things, such as independence, and take no other information into account (such as how well you’ve studied for the interviews, how qualified you are for the position, etc). Those numbers are just the amount of resumes they received, from all walks of life and all parts of the world, rather than the number of the people who actually managed to interview for the position. When you start taking those figures into account, particularly when you’ve properly studied the domain of coding interviews, the odds start looking a lot more achievable, and Stanford or Harvard start looking a lot harder to get into by comparison (whether or not they actually are, as similar factors are at work in their quoted acceptance rates as well).

The interesting truth is that if you were able to master a majority of the easy problems on websites like LeetCode or in popular interview texts, and if you were able to land an offer at some top company (Amazon, Microsoft, etc), your odds of getting into any other one of those companies vastly increase. Some of these companies act as ladders. In fact I, the author, was not interesting enough to interview for and make it through the rather specialized and quite intense 30 something person, panel style interviews that I faced at Apple during the Steve Jobs era, until I had first successfully made it into Amazon and had the opportunity to do some interesting things there. That's not to say that Amazon enabled me to get in there, but that the knowledge, study, and lessons learned in both getting in and working at Amazon, including having earned a Master's degree in between my two attempts, easily explained away many of the factors involved. In fact, my first Google on-site interview and plane-ticket to Mountain View even came the same exact day I received an Amazon offer (that's another possibly interesting story for a different time).

As we’ll see later in the book, assuming you’re good enough to interview for and be hired by any of the top tech companies, your odds might be closer to 25% than 1 in 500. That simple statement implies that interviewing 4 times should land you with at least 1 offer, at those odds, statistically speaking. In practice it may not be that simple by itself—you should be improving your interview skills and algorithms + data structures knowledge each time—but this process and these numbers assume that you are already doing that anyways.

The Blackstone Ratio

Having said that the odds favor the house, let me give some more information about this. Interviews at many top tech companies seem to be set up such that there is a bias towards precision rather than recall; if you’re not familiar with those terms, which appear in statistics and machine learning in particular, they mean that the companies would rather reject, say, 4 good candidates rather than hire a single bad one. And that’s because the cost of one bad hire far outweighs the benefits of hiring 4 good candidates—or at least, that sounds good on paper; in practice most people would disagree with what this optimal ratio actually is or even what it should be.

My own personal experience, although anecdotal and without even a shred of supporting evidence other than personal experience, seems to suggest that this ratio is close to 1 in 4 on average. This should also imply that for every 4 top companies you conduct an on-site interview at (assuming you've made it this far), you should expect to see at least 1 offer. This doesn't count technical phone screens or recruiter screens, but just those where a company believed enough in you to invest in a plane ticket to bring you on-site, or otherwise spend a valuable day of their employees' time investigating your potential, via the on-site interview process.

Perhaps some interview candidates seem to land an offer in one shot for any given company, and so their personal ratio is 1, although as an interviewer, perhaps that one and done magical candidate only has a 1 in 10 acceptance rate for other candidates that they themselves interview, implying their bar is much higher than what they themselves experience. This would then raise the question of whether the interview questions they give to others are properly calibrated? Is 1 in 10 too strict? Is it not strict enough? What if there are simply that many good candidates out there, shouldn't all 10 come through? Well, getting 10 heads on a coin in a row is possible, but one might expect that this is simply variance about the mean, and the true odds of a coin flip are closer to 50/50. Interviewing is of course not a coin flip, and as each of us has seen and felt, the odds are quite more pessimistic than that. And also, each person who reads this likely has a different opinion and experience here for what this ratio truly is, so please comment sometime and let me know what your personal experience has been here.

This concept should also sound familiar because it forms the basis for an important tenant of the U.S. judicial system, where we have the concept of the “Blackstone ratio.” This ratio suggests that it’s a far better outcome that, say, even 10 guilty persons should go free before a single innocent person sits in jail for some crime that they did not commit. This is an example of how favoring precision (for instance, only giving an affirmative on a decision when you’re really, really sure, out of fear of being wrong, and out of the cost of being wrong) over favoring recall (essentially, guessing a lot and not really being sure, out of a fear of missing out on what could be; you'll make a lot of wrong guesses, but you'll catch, say, a few that would otherwise have been missed if you didn't). The precise, mathematical definitions of precision and recall are of course different from this, but I use the above analogies to communicate the idea of what these concepts mean.

The argument for a bias towards precision, e.g. only hiring candidates that you are absolutely sure about, suggests that the cost of one bad hire implies a team will become heavily disrupted by this person's presence. It might also suggest that it would then take multiple teammates to go in and "clean up" after that person, thereby lowering the productivity of the team if they hadn't been hired at all, incurring a cost that each teammate must bare. Whether or not that is actually the case, it illustrates a feeling that many people have about this. It also speaks to the idea that good teammates only want to work with other good teammates, and if a "bad" one gets into the mix, the "good" ones might find their way to another team, thus destroying the team over time from within.

In fact, it’s often said, including by Steve Jobs on notable occasions, that A players hire other A players, because they only want to work with A players. A players don’t want to work with B players, and B players are intimidated by A players, so they tend not to hire them—they may end up hiring a C player, in fact, so that they can feel good about their own skills, or because they don’t care. Either way, without an Amazon style attitude of “raising the bar” with each new hire, or at least without keeping the bar consistently high over time, this kind of attitude can slowly begin to destroy the company from inside, diluting its pool of talent.

Or at least, this is a good enough story that is told to justify the process, but it can sure feel like something is wrong when you’re on the wrong end of it—particularly when you are good enough to be hired by otherwise equal competition, offer in hand, and you get rejected by this process for another, probably your preferred choice of company, for unknown or seemingly arbitrary reasons. The truth is probably somewhere in between, but regardless of the outcome, there are typically many areas of improvement that you can make, to prove to others what you probably already know--that you're talented enough to get an offer. But to do that, you must remain persistent, accepting the process for what it is, an error-prone system biasing heavily towards precision. You then need to commit yourself to continue growing until the evidence of hire is overwhelming and obvious to your interviewers. And then, one fine morning, that magical offer drops, and suddenly every rejection that came before no longer matters. Only the thrill of accomplishment remains.

This is all here to inform you and remind you that, based on my own personal experience of getting offers at and working for many top companies including Amazon, Apple, and Microsoft, interviewing multiple times is not only common, but absolutely necessary in most cases. Those who really want to make it in will try, try, try again. The odds are you will likely make it in at some point if you are good enough and persistent enough, analyzing and acting on your failures, turning what seems like a losing battle into a fighting chance, as great fighters and artists like you probably do, and engaging in good old fashioned Malcom Gladwell style 'deliberate practice'.."

[1] Nisen, M. (2014). “Here’s why you only have a 0.2% chance of getting hired at Google.” Quartz. Accessed 11/28/2020, https://qz.com/285001/heres-why-you-only-have-a-0-2-chance-of-getting-hired-at-google/

Unpublished work. ? Copyright 2020, Joseph Johnson Jr.