Interview Brain Freeze: a Self-experiment

Clueless Beginnings

My first interview was in 2000, just after graduating. Since I had just obtained my Bachelor in one of the hot topics of the time, and from the hottest university in the country, it was obvious to me I would simply be invited in, based on pedigree alone. After all, I had put a lot of effort into it, with the grades and projects to show it. That was not obvious to my interviewers, who sent me off after a few short minutes. Since I did not really care about the job, I decided my interviewers were clueless, and lived on.

Preparation = Prediction

During the years, I got burned on a number of occasions, those on which I actually had to think, instead of already knowing. Since those were the occasions pertaining to the more interesting jobs, I wised up and decided not to leave interviews to chance any longer. This was before sites like glassdoor.com existed, and way before sites like leetcode.com, careercup.com, geeksforgeeks.org. My process was simple: use the history of questions I have been asked, augment those with web search for any additional questions, filter that relative to the company and position at question, and boost that according to knowledge of the individual people who were going to interview you.

It Works

My best two successes were crafting a two year plan to get employed at a top AAA studio (which worked), and receiving the most paradoxical comment I ever did post-interview from another. The latter went as follows: My interviewer was someone with an internet presence, and a bit of an authority on a niche subject related to game technology. It was therefore easy to predict most of his questions. By that time, twitter existed, and being someone who liked to display his superiority publicly by complaining about people not knowing the details of this or that technical matter that was crucial (to him...), mining a list of such matters was a thing quickly achieved. At the time, I thought of bit-twiddling (https://graphics.stanford.edu/~seander/bithacks.html) exercises as useless and merely annoying, but having been able to predict exactly the bit-twiddling question I was asked, I was happy, and gave my answer. I framed my answer such that it would seem that I had never seen the topic before, and that I solved it on the fly using heuristics, creative thinking, and a faint memory about a fact from university days. The comment that I later received was that I did not pass the interview, despite my amazing answer to the bit-twiddler, I was not getting hired.

The Brain Freeze

I mentioned I was burned on a few occasions. I realised that I indeed cannot (truly) think while talking. Whenever I had to do that I transformed into an idiot. If you really want me to think, send me into the shower (a phenomenon familiar to many). Look up 'Brain Freeze' to see how commonly this problems occurs during interviews. The only tip you will read is to prepare, prepare again, and then once more. But wait a minute. Why do you have to do that? Prepare for what exactly? For showing that you can do what you already do eight hours per day? What is wrong with my naive tactic of pedigree alone? Is not the best indicator of future success doing task X, the amount of success you had doing task X in the past? Why is solving problems you never have to solve on the job, in a format you never use (whiteboard coding), together with people you never met, a better indicator? During the majority of my hours on the job, I have done one of two things:

1. Solve problems silently, alone, in my comfort zone, and
2. Explain my solutions, vocally and pictorially, to a group.

The usual interview format is nothing like it. The 'idea' is that you cannot predict the questions (which by now everyone can), and then proceed to solve them vocally, on a whiteboard, with a group.

Post-hoc Bias

I have had several discussions about this, and some people truly believe that when interviewing in the currently standard ways, they are obtaining a good measure of someone's future performance, a feat no brain-expert, psychoanalyst and future-teller combined can achieve in multiple months. There is nothing that supports this theory, except for post-hoc bias of the form: "Look, all these people that I already hired this way, they have been a total success, it was almost always the right decision", or "Our company is doing great, so our interviewing is just fine". Short of going into how wrong this is from the point of view of statistical inference, it clearly suffers from at least two basic biases. The first is amplifying what worked and ignoring what did not, the second being that you have nothing to compare against: it is not like you experimented with half of the candidates as a control group that you hired based on basic history check and then coin tossing (a hiring technique that I am sure would prove my point).

The Proportionally-Tuned Filter

The reality of the matter is totally different. The 'difficulty' of an interview is in direct relation with the number of candidates to filter out. This is why companies that are highly desirable, in time, end up sharpening their tests so that more or less, they filter the one per thousand they need. The less than desirable ones on the other hand, end up loosening their interviews, such that they can hopefully fill some of the openings they have. The rest is nothing more than pretentious games, triggering millions of 'Brain Freeze' events per year if not billions.

Self-experiment: I am a 40 year old veteran, and I can't do it

Finally to the self-experiment. I went and solved one function problem of each of the difficulties: easy, medium, hard on geeksforgeeks.com. I did it in my usual setting. Alone, silent. Then I went for the popular 'egg dropping' problem, which according to geeksforgeeks.org is asked by all of Amazon, D-E-Shaw, Goldman-Sachs, Google, Hike, MakeMyTrip, MAQ-Software, Myntra, Nearbuy, Opera, Oracle, Philips, Samsung, Service Now, Unisys and VMWare. I know this problem because a colleague once decided to spice up our interview process with it, since he felt we needed to hire people at least as clever as us, and since clever companies seemed to all use it, it would be clever for us to use it as well (this is one way in which these idiocies propagate). I remember that after the interview (I take part in all of our interviews), I thought about it during the train ride home. I reached the conclusion that a prerequisite for solving it was realising that on each drop, both the egg breaking and not breaking should have the same worst case outcome, for if not, the worst case would be worse if one event happened, and better if another happened, and would not be the worst case. Then I forgot about the problem until this self-experiment.

Deciding to simulate a whiteboard-coding interview, I wondered how fast I would reach the same conclusion while talking loudly, and drawing pictures. I did not 'brain freeze' because no one was staring at me, but what I produced was a long series of brain-dead sketches, and 70% of what I said was going round in circles, a big part of it was keeping forgetting that eggs break in the higher part of the building and don't break in the lower and not vice versa. Basically, I had achieved total idiocy. Despite patience, a relatively comfortable setup, time passed and my old conclusion was nowhere in sight. If you can't solve a problem, try to solve an easier one. I went back and tried the medium function problem, but instead of coding it, I decided to try to whiteboard-solve it, talking loudly again, and I failed. Now I am sure this skill is learnable, but for the record, right now I do not have it. In other words, so far, in all of my career, which was not shabby if you ask around, I did not need it.

The Forgetful Candidate

Role-playing a helpful interviewer, I peaked a bit into the 'egg dropping' problem and realised that my conclusion that 'both the egg breaking and not breaking should have the same worst case outcome' was nothing but the optimal sub-structure property in dynamic programming, the discipline to which such problems belong. I then remembered that I did deal with dynamic programming when I was learning reinforcement learning in order to accelerate footstep-based motion planning (https://www.youtube.com/watch?v=f3V8yX8_UXY). How could I have done this without knowing about the substructure property and detecting it in the egg dropping problem? Memory of course. I have done so many things between then and now, and my brain simply jettisons, to my dismay, many things that would otherwise be useful to keep. I say this because of another common problem that occurs during interviews.

People are questioned about the details of something they did in the past, and they often fail, appearing to have treated it in a shallow manner, which is not true. I am often engrossed in a problem, for a number of days or weeks, and when I later re-read the notes I have written, I weep from the pain of how much I have already forgotten. On that problem, I feel like I was an expert compared to my current totally blank self. All is gone. But this is nothing but the artefact of the kinds of jobs some of us do. If you had to solve forty quite different problems each year, each requiring your full concentration and memory, it is not surprising that you forgot, sometimes fully, the details of problems even six weeks ago. This effect is stronger when you intrinsically are not passionate about the problem, but that should not be taken against you, on the contrary! You focused on a problem intrinsically not interesting to you and you did solve it successfully. That is the goal after all.

I Will Flip Coins

I guess my point is not even a point. We all know the technical interview is an excuse for filtering. If (when) I will have enough decision power over such topics, I promise you that I will thoroughly and with all zeal check the person's profile and history, ask them to come loaded with details about what they did, and to take multiple days full-time to prepare those details if they did not have them already, because that is what our getting-together will focus on. To detect if they are lying, which is something companies worry about, I would thoroughly talk with people they worked with, asking pointed questions, instead of the soft questions they are usually asked. For the best indication of future performance, is past performance in similar conditions. Is that costly? Maybe, but it's the right thing to do. Having done that, I might very well end up with too many good candidates for the job. At that point, I will inform all of them that they would be hired had they been the only candidate, and that they will receive the results of the lottery as soon as I have flipped all the coins.

Until then, see you at the next freeze...

UPDATE

A continuation of this article, with a solution to the 'egg drop problem' can be found here: https://www.dhirubhai.net/feed/update/urn:li:activity:6346428153963126785