How an Interviewer Scorecard Has Increased the Quality of Feedback and Reduced the Time to Hire

Everyone in talent acquisition seems focused on interview questions, but the real art of interviewing is in knowing what to make of the answers, to quickly determine if a candidate has the insight and aptitude to be a success in your open position.

The Product team at LinkedIn recently set out to understand who their star interviewers are (and how to develop more of them) to elevate the quality of incoming talent and to reduce the time to hire. As part of an overhaul of their hiring process for product managers, they wanted to track which of their interviewers provided feedback that was timely, helpful, and on-target.

Amy Schultz, head of recruiting for Product, and Caroline Gaffney, Product’s chief of staff, worked with Katie Sittler of LinkedIn’s People Analytics team last year to develop a scorecard to assess interviewers.

The scorecard and the new hiring process have given the Product team a consistent bar to measure talent, improved the candidate experience, and cut time to hire in half.

The Interview Scorecard captures a team member’s candidate evaluations and compares them to their peers’

In Product’s new hiring process, each candidate goes through six interviews — a recruiter phone screen, a technical phone screen, and four onsite interviews. Each onsite interview lasts 30 minutes and is focused on a different core competency that Product has identified as central to being a successful product manager at LinkedIn:

1. builds quality products that members and customers want
2. takes initiative and gets stuff done
3. strong communicator, collaborator, and leader
4. mission-driven, ecosystem thinker

Interviewers are then asked to fill an evaluation form that asks for four bits of feedback:

1. Rate the candidate on a scale of 1 to 5, with 5 being the most enthusiastic rating.
2. Would you hire this person for this position? Yes or no?
3. List the questions you asked.
4. Provide an assessment of the candidate’s strengths and weaknesses.

By collecting the questions that each interviewer asks, the Product team hopes eventually to match those questions with performance evaluations to see which ones do the best job of predicting success. The assessments of strengths and weaknesses are tied to answers each candidate gives and, when done well, are what Ryan Roslansky, the SVP of Product, and the hiring committee find most useful.

The answers to the first two questions, on the other hand, help shape the scores that interviewers receive from People Analytics. Here are the basic pieces of information captured in the Interviewer Scorecard:

1. Employee name and organization
2. Number of interviews
3. Number of interviews for which feedback was completed
4. Percentage of interviews for which feedback was completed
5. Average rating (all candidates)
6. Average rating (hired candidates)
7. Rating distribution (all candidates)
8. Rating distribution (hired candidates)
9. Average number of interviews per week
10. Number of candidates interviewed who were hired

The rating distributions might, for example, show that an interviewer’s scores are almost always 2s and 3s, so that when a 4 or a 5 pops up, the hiring committee knows to pay particular attention. 

Here is a mock-up of an interviewer scorecard that features columns for the data that LinkedIn’s Product team captures on its version. This link will take you to version of the scorecard that you can copy and share with your team.

Amy says that future scorecards will also include a kind of “batting average” that captures the percentage of times the interviewer’s Yes or No on whether to hire the candidate matches the final decision of the hiring committee. 

The Product team is also looking to roll out a phase 2 scorecard that will link quality of hire back to individual interviewers. So, over the next two review cycles, they will start matching the performance of recent hires with the particular team members who interviewed them as candidates.

An interviewer scorecard allows talent acquisition to see who does a great job — and who needs a bit of additional coaching

There is an old business axiom that what you measure is what you’ll get. And that has certainly proven true for LinkedIn’s Product team — in two important ways.

First, since they’ve started keeping track of what percentage of their interviews receive completed evaluation forms, the number has skyrocketed. In the most recent measuring period, 93% of the time interviewers completed the evaluations. Some of that increase is due to a simplified evaluation form and a new interview training course, but some of it simply comes from making it a teamwide metric.

Second, by measuring interviews and keeping track of performance, the Product team has sent a clear message that this work is highly valued and worth celebrating. Laurels currently come in the form of grateful notes from Ryan. But Amy is eager to launch a more formal recognition program, with perhaps a quarterly award to celebrate Product’s star interviewers.

“By recognizing our top interviewers,” Amy says, “more people get involved in interviewing and that speeds up the process.” Since the new hiring process rolled out and the scorecards were introduced, the number of interviewers has grown from 57 to over 100 — and the average time to “approved for hire” has been cut from 83 days to 41 and is still dropping.

The scorecard also helps the interviewers. “They actually want to know if they’re a good interviewer or not,” Amy says. “They’ve asked us to give them some metrics, so that’s also why we’ve built this scorecard.”

The metrics create transparency into the process, helping to identify interviewers who are struggling and would benefit from more coaching as well as interviewers who are nailing it and should get a shout-out.

Final thoughts

We are a society that loves metrics. We turn to Yelp to get a take on restaurant choices and Rotten Tomatoes to assess our movie options. We have more sophisticated metrics to understand everything from traffic patterns to athletic performance.

So . . . why no interview metrics?

Some companies still see interviews as an intuitive process that requires art, magic, and improvisation to get it right. But by changing your view on interviewing and moving to a defined process with assigned roles and a standard script of questions, you can start to measure interviewer performance and make apples-to-apples comparisons.

And given all it has done since it rolled out earlier this year, the Interview Scorecard has earned a 5.

To receive blog posts like this one straight in your inbox, subscribe to the blog newsletter.